CWLProv: needs to add running the conformance test to regular testing

195 tests passed, 2 failures, 0 unsupported features

[ ] add ./run_test.sh RUNNER=cwltool EXTRA="--provenance $PWD/RO_test" to ci.commonwl.org or maybe seq 1 133 | parallel ./run_test.sh RUNNER=cwltool EXTRA=\"--provenance $PWD/ro_test/RO_{#}\" -n{#} ? Bonus: Check packed workflows for any file:/// references that slipped through
```
find ro_test/  -name packed.cwl  | xargs grep "file://$PWD"
find ro_test/  -name packed.cwl  | xargs grep path\":
```

[x] 87 "Test directories in secondaryFiles"
[x] 136 "Test specifying secondaryFiles in subdirectories of the job input document."

[x] 137 "Test specifying secondaryFiles in same subdirectory of the job input as the primary input file."

Traceback (most recent call last):                             
File "/home/mcrusoe/cwltool/cwltool/executors.py", line 167, in run_jobs
job.run(runtime_context)                                                                 
File "/home/mcrusoe/cwltool/cwltool/job.py", line 498, in run                                                                                       
self._execute(runtime, env, runtimeContext)                                                                                                       
File "/home/mcrusoe/cwltool/cwltool/job.py", line 332, in _execute                                                                                  
outputs, runtimeContext.process_run_id, str(self.name))                                                                                            
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 839, in generate_output_prov                                                               
entity = self.declare_artefact(value)                                      
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 736, in declare_artefact                                                                   
(entity, _, _) = self.declare_file(value)
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 571, in declare_file                                                                       
(sec_entity, _, _) = self.declare_file(sec)
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 514, in declare_file                                                                       
raise ValueError("Must have class:File: %s" % value)                                                    
ValueError: Must have class:File: {'basename': 'input.txt_idx8', 'location': 'file:///tmp/tmpvswaqxeb/input.txt_idx8', 'class': 'Directory'}          
Workflow error, try again with --debug for more information:                              
Must have class:File: {'basename': 'input.txt_idx8', 'location': 'file:///tmp/tmpvswaqxeb/input.txt_idx8', 'class': 'Directory'}

[x] 61 "Test parameter evaluation, no support for JS expressions"
[x] 62 "Test parameter evaluation, with support for JS expressions"
[x] 73 "Test record type output binding."
[x] 143 "Test record type inputs to and outputs from workflows"
[x] 160 "Test Any parameter with record input to a tool"
[x] 165 "Test Any parameter with record input to a tool" (in a workflow) ("@id": "urn:uuid:8238136c-4736-44a2-a6cc-16fbc789fdd8" in the output records

[x] 90 "Test file literal as input"
[x] 120 "Test file literal as input without Docker"
[x] 190 "Pipe to stdin from literal File via a Directory literal"

[x] 191 "Test non-stdin reference to literal File via a Directory literal"

/home/mcrusoe/cwltool/env3/bin/cwltool 1.0
Resolved 'v1.0/cat3-tool.cwl' to 'file:///home/mcrusoe/common-workflow-language/v1.0/v1.0/cat3-tool.cwl'
Got workflow error
Traceback (most recent call last):
File "/home/mcrusoe/cwltool/cwltool/executors.py", line 164, in run_jobs
runtime_context.research_obj)
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 459, in evaluate
self.used_artefacts(customised_job, self.workflow_run_uri)
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 821, in used_artefacts
entity = self.declare_artefact(value)
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 736, in declare_artefact
(entity, _, _) = self.declare_file(value)
File "/home/mcrusoe/cwltool/cwltool/provenance.py", line 531, in declare_file
with fsaccess.open(location, "rb") as fhandle:
File "/home/mcrusoe/cwltool/cwltool/stdfsaccess.py", line 41, in open        
return open(self._abs(fn), mode)                    
FileNotFoundError: [Errno 2] No such file or directory: '_:221ad84f-daa1-48b7-bfee-8f28dae22ed8'                                                       
Workflow error, try again with --debug for more information:
[Errno 2] No such file or directory: '_:221ad84f-daa1-48b7-bfee-8f28dae22ed8'

[x] 118 "Test empty writable dir with InitialWorkDirRequirement inside Docker"
[x] 55 "Test InitialWorkDirRequirement linking input files and capturing secondaryFiles on input and output. Also tests the use of a variety of parameter references and expressions in the secondaryFiles field. " size: 0 missing, maybe a serialization error?

From CWL v1.1 conformance tests

[ ] 234 "symlink to file inside of working directory should be retrieved"

Traceback (most recent call last):
File "/home/michael/cwltool/cwltool/executors.py", line 188, in run_jobs
job.run(runtime_context)
File "/home/michael/cwltool/cwltool/job.py", line 693, in run
self._execute(runtime, env, runtimeContext, monitor_function)
File "/home/michael/cwltool/cwltool/job.py", line 366, in _execute
outputs, datetime.datetime.now())
File "/home/michael/cwltool/cwltool/provenance.py", line 487, in record_process_end
self.generate_output_prov(outputs, process_run_id, process_name)
File "/home/michael/cwltool/cwltool/provenance.py", line 826, in generate_output_prov
entity = self.declare_artefact(value)
File "/home/michael/cwltool/cwltool/provenance.py", line 715, in declare_artefact
(entity, _, _) = self.declare_file(value)
File "/home/michael/cwltool/cwltool/provenance.py", line 509, in declare_file
relative_path = self.research_object.add_data_file(fhandle)
File "/home/michael/cwltool/cwltool/provenance.py", line 1425, in add_data_file
self._add_to_bagit(rel_path, sha1=checksum)
File "/home/michael/cwltool/cwltool/provenance.py", line 1486, in _add_to_bagit
raise IOError("File %s does not exist within RO: %s" % (rel_path, local_path))
OSError: File data/cd/cd28ec34f3f9425aca544b6332453708e8aaa82a does not exist within RO: /tmp/opj488vu/data/cd/cd28ec34f3f9425aca544b6332453708e8aaa82a

[ ] 236 "inplace update has side effect on directory content"

Compare failure expected: {
"a": [
    {
        "basename": "blurb",
        "class": "File",
        "location": "blurb"
    }
],
"b": [
    {
        "basename": "blurb",
        "class": "File",
        "location": "blurb"
    }
]
}
got: {
"a": [],
"b": [
    {
        "basename": "blurb",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "class": "File",
        "location": "file:///tmp/tmpo9kr4uf3/blurb",
        "path": "/tmp/tmpo9kr4uf3/blurb",
        "size": 0
    }
]
}

[ ] 242 "Test that default behavior is 'no_listing' if not specified"
[ ] 243 "Test that 'listing' is not present when LoadListingRequirement is 'no_listing'"
[ ] 244 "Test that 'listing' is not present when loadListing on input parameter is 'no_listing'"
[ ] 245 "Test that 'listing' is present in top directory object but not subdirectory object when LoadListingRequirement is 'shallow_listing'"
[ ] 246 "Test that 'listing' is present in top directory object but not subdirectory object when loadListing on input parameter loadListing is 'shallow_listing'"
```
Compare failure expected: {
"out": true
}
got: {
"out": false
}
```

common-workflow-language / cwltool

CWLProv: needs to add running the conformance test to regular testing #916