YWConfiguration issue

[sycao5]

I ran into an issue about yw configuration. My code is attached below.

 % Use yw.properties for configuration                                     
  config = YWConfiguration();
 % Generate YW.Process_View dot file                   
  config.applyPropertyFile(runManager.PROCESS_VIEW_PROPERTY_FILE_NAME); % Read from process_view_yw.properties
  gconfig = config.getSection('graph');

This code can read the yw.properties in the unit test of matlab. The debug information is attached below too.

  Name        Size            Bytes  Class                                     Attributes

  config      1x1                org.yesworkflow.config.YWConfiguration             

  Name         Size            Bytes  Class                Attributes

  gconfig      1x1               java.util.HashMap              

But when I run in the command-line mode of matlab, the code starts to work strangely. The debug information is attached here as well.

  Name        Size            Bytes  Class                                     Attributes

  config      1x1                org.yesworkflow.config.YWConfiguration              

  Name         Size            Bytes  Class     Attributes

  gconfig      0x0                 0      double        

This error caused the yw.properties file cannot be used for configuration. Would you like to take a look at the class YWConfiguration again and troubleshooting the possible problem?

Thank you very much!

[sycao5]

function captureProspectiveProvenanceWithYW(runManager, runDirectory)
            % CAPTUREPROSPECTIVEPROVENANCEWITHYW captures the prospective provenance using YesWorkflow 
            % by scannning the inline yesWorkflow comments.

            import java.io.BufferedReader;
            import org.yesworkflow.annotations.Annotation;
            import org.yesworkflow.model.Program;
            import org.yesworkflow.model.Workflow;
            import java.io.FileInputStream;
            import java.io.InputStreamReader;
            import java.util.List;
            import java.util.HashMap;
            import org.yesworkflow.config.YWConfiguration;

            try
                % Read script content from disk
                in = FileInputStream(runManager.execution.software_application);
                reader = BufferedReader(InputStreamReader(in));

                % Use yw.properties for configuration                                     
                config = YWConfiguration();

                % Call YW-Extract module
                runManager.extractor = runManager.extractor.reader(reader); 
                annotations = runManager.extractor.extract().getAnnotations();

                % Call YW-Model module
                runManager.modeler = runManager.modeler.annotations(annotations);
                runManager.modeler = runManager.modeler.model();
                runManager.workflow = runManager.modeler.getModel().program;

                % Call YW-Graph module
                if runManager.configuration.generate_workflow_graphic
                    import org.yesworkflow.graph.GraphView;
                    import org.yesworkflow.graph.CommentVisibility;
                    import org.dataone.util.HashmapWrapper;
                    import org.yesworkflow.graph.LayoutDirection;

                    % Set the working directory to be the run metadata directory for this run
                    curDir = pwd();
                    curDir
                    runManager.PROCESS_VIEW_PROPERTY_FILE_NAME

                    runManager.grapher = runManager.grapher.workflow(runManager.workflow);

                    % Generate YW.Process_View dot file                   
                    config.applyPropertyFile(runManager.PROCESS_VIEW_PROPERTY_FILE_NAME); % Read from process_view_yw.properties
                    gconfig = config.getSection('graph');
                    runManager.processViewDotFileName = gconfig.get('dotfile');
                    runManager.grapher.configure(gconfig);
                    runManager.grapher = runManager.grapher.graph();           

                    % Generate YW.Data_View dot file                  
                    config.applyPropertyFile(runManager.DATA_VIEW_PROPERTY_FILE_NAME); % Read from data_view_yw.properties 
                    gconfig = config.getSection('graph');
                    runManager.dataViewDotFileName = gconfig.get('dotfile');
                    runManager.grapher.configure(gconfig);
                    runManager.grapher = runManager.grapher.graph();

                    % Generate YW.Combined_View dot file                   
                    config.applyPropertyFile(runManager.COMBINED_VIEW_PROPERTY_FILE_NAME); % Read from comb_view_yw.properties
                    gconfig = config.getSection('graph');
                    runManager.combinedViewDotFileName = gconfig.get('dotfile');
                    runManager.grapher.configure(gconfig);
                    runManager.grapher = runManager.grapher.graph();

                    % Create yesWorkflow modelFacts prolog dump 
                    import org.yesworkflow.model.ModelFacts;
                    import org.yesworkflow.extract.ExtractFacts;

                    cd(runDirectory); 
                    pwd

                    modelFacts = runManager.modeler.getFacts();  
                    gconfig = config.getSection('model');
                    runManager.mfilename = gconfig.get('factsfile');
                    fw = fopen([runDirectory filesep runManager.mfilename], 'w'); 
                    if fw == -1, error('Cannot write "%s%".',runManager.mfilename); end
                    fprintf(fw, '%s', char(modelFacts));
                    fclose(fw);

                    % Create yesWorkflow extractFacts prolog dump
                    extractFacts = runManager.extractor.getFacts(); 
                    gconfig = config.getSection('extract');
                    runManager.efilename = gconfig.get('factsfile');
                    fw = fopen([runDirectory filesep runManager.efilename], 'w');    
                    if fw == -1, error('Cannot write "%s%".',runManager.efilename); end
                    fprintf(fw, '%s', char(extractFacts));
                    fclose(fw);

                    cd(curDir); % go back to current working directory          
                end  

            catch ME 
                error(ME.message);
            end      
        end

[sycao5]

Running org.dataone.client.run.RunManagerTest

In testYesWorkflow() ...

curDir =

/Users/syc/Documents/matlab-dataone

ans =

/Users/syc/Documents/matlab-dataone/test/resources/yw_process_view_7.properties

================================================================================
Error occurred in org.dataone.client.run.RunManagerTest/testYesWorkflow and it did not run to completion.

    --------------
    Error Details:
    --------------
    Error using org.dataone.client.run.RunManager/captureProspectiveProvenanceWithYW (line 281)
    Attempt to reference field of non-structure array.

    Error in org.dataone.client.run.RunManager/callYesWorkflow (line 1031)
                    runManager.captureProspectiveProvenanceWithYW(dirFullPath);

    Error in org.dataone.client.run.RunManagerTest/testYesWorkflow (line 136)
                testCase.mgr.callYesWorkflow(scriptPath, testCase.mgr.execution.execution_directory);

[sycao5]

The issue is caused by the path problem. The original implementation is depending on the working directory to be src/

Solution: use matlab_toolbox_directory to build the path from so a default config file gets set if the user doesn’t provide one. Move all yw.properties to lib/yesworkflow/

[sycao5]

The problem is still there in the command-line

[tmcphillips]

Because you're using the YWConfiguration class directly, applyPropertyFile() needs to throw an exception when a configuration file referred to in a call to it doesn't exist. This would have saved you a lot of time looking for the bug!

Right now YWConfiguration.applyPropertyFile() fails silently when the referenced file doesn't exist so that the yw.properties file is effectively optional when using the CLI. I'll change the CLI to check for yw.properties and then call applyPropertyFile() only if it exists.

[tmcphillips]

I just put in checks for invalid parameters and nonexistent files in each of the public methods of YWConfiguration. I've also updated YesWorkflowCLI to check for the config files before calling methods on YWConfiguration so that exceptions aren't thrown when the configuration files are not present (YW views configuration files as optional).

I'm now adding unit tests and Javadoc to the public methods of YWConfiguration.

I'll let you know when I've pushed all of these changes. However, I recommend sticking with the version of YW currently used with Matlab until after any important demos that may be coming up in the short term.

[sycao5]

Many thanks! Have a good night!