search

common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
336 stars 231 forks source link

Error while running bcbio workflow #828

Open FarahZKhan opened 6 years ago

FarahZKhan commented 6 years ago

Expected Behavior

The workflow should execute successfully creating RO.

Actual Behavior

Workflow fails with the following error: Anonymous file object must have 'contents' and 'basename' fields.

Workflow Code

https://github.com/bcbio/test_bcbio_cwl/blob/master/somatic/somatic-workflow/main-somatic.cwl

Full Traceback

(testing) farahk@4180L-142848-M:~/Documents/conformanceTest-CWLProv/test_bcbio_cwl/somatic/somatic-workflow$ cwltool --provenance ROtest main-somatic.cwl main-somatic-samples.json 
/Users/farahk/Documents/testing/bin/cwltool 1.0.20180702035104
[provenance] Temporary research object: /var/folders/bs/8fyl2nt50252htp1d2ct1rd1g51vtc/T/tmp0f6876w5
Resolved 'main-somatic.cwl' to 'file:///Users/farahk/Documents/conformanceTest-CWLProv/test_bcbio_cwl/somatic/somatic-workflow/main-somatic.cwl'
[provenance] Added packed workflow: workflow/packed.cwl
steps/alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/prep_align_inputs.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/process_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/process_alignment.cwl:83:3: Unknown hint http://arvados.org/cwl#APIRequirement
steps/merge_split_alignments.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/prep_samples_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/prep_samples.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/postprocess_alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/postprocess_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/postprocess_alignment.cwl:54:3: Unknown hint http://arvados.org/cwl#APIRequirement
steps/combine_sample_regions.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/combine_sample_regions.cwl:42:3: Unknown hint http://arvados.org/cwl#APIRequirement
steps/batch_for_variantcall.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/get_parallel_regions.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/variantcall_batch_region.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/variantcall_batch_region.cwl:105:3: Unknown hint http://arvados.org/cwl#APIRequirement
steps/variantcall_batch_region.cwl:106:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
steps/concat_batch_variantcalls.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/concat_batch_variantcalls.cwl:42:3: Unknown hint http://arvados.org/cwl#APIRequirement
steps/postprocess_variants.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/compare_to_rm.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/summarize_vc.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/qc_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/pipeline_summary.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
steps/pipeline_summary.cwl:76:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
steps/multiqc_summary.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
[provenance] Finalizing Research Object
[provenance] Generated bagit metadata: /var/folders/bs/8fyl2nt50252htp1d2ct1rd1g51vtc/T/tmp0f6876w5
[provenance] Deleting existing /Users/farahk/Documents/conformanceTest-CWLProv/test_bcbio_cwl/somatic/somatic-workflow/ROtest
[provenance] Research Object saved to /Users/farahk/Documents/conformanceTest-CWLProv/test_bcbio_cwl/somatic/somatic-workflow/ROtest
Traceback (most recent call last):
  File "/Users/farahk/Documents/testing/bin/cwltool", line 11, in <module>
    sys.exit(main())
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/main.py", line 574, in main
    secret_store=runtimeContext.secret_store)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/main.py", line 221, in init_job_order
    basedir=file_uri(str(input_basedir) + "/"))
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/main.py", line 288, in printdeps
    {"$include", "$schemas", "location"}, loadref)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/process.py", line 933, in scandeps
    r.extend(scandeps(base, v, reffields, urlfields, loadref, urljoin=urljoin))
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/process.py", line 936, in scandeps
    r.extend(scandeps(base, d, reffields, urlfields, loadref, urljoin=urljoin))
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/process.py", line 939, in scandeps
    normalizeFilesDirs(r)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/pathmapper.py", line 82, in normalizeFilesDirs
    visit_class(job, ("File", "Directory"), addLocation)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/utils.py", line 234, in visit_class
    visit_class(d, cls, op)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/utils.py", line 231, in visit_class
    visit_class(rec[d], cls, op)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/utils.py", line 234, in visit_class
    visit_class(d, cls, op)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/utils.py", line 229, in visit_class
    op(rec)
  File "/Users/farahk/Documents/testing/lib/python3.6/site-packages/cwltool/pathmapper.py", line 58, in addLocation
    raise validate.ValidationException("Anonymous file object must have 'contents' and 'basename' fields.")
schema_salad.validate.ValidationException: Anonymous file object must have 'contents' and 'basename' fields.

Your Environment

mr-c commented 6 years ago

I get an error running this without --provenance

[job compare_to_rm_3] Using cached output in /home/mcrusoe/cwltool/bcbio_cache/0cc080e1df32f35ed64162e045488fc4
Workflow error, try again with --debug for more information:
Output file path /var/lib/cwl/stg8dc33acf-dc9f-4d14-925c-38c981e754e5/b1-analysis_blocks.bed must be within designated output directory (/var/spool/cwl) or an input file pass through.
mr-c commented 6 years ago

Perhaps a bug with --cache

Raw output from /home/mcrusoe/cwltool/bcbio_cache/fe7d3920d9ec92d976f759ec567a0f63/cwl.output.json: {                                                 
    "vc_rec": [                                                                                                                                       
        {                                                                                                                                             
            "analysis": "variant",                                                                                                                    
            "batch_samples": [                                                                                                                        
                "Test2",                                                                                                                              
                "Test1"
            ],                                                                                                                                        
            "config__algorithm__callable_regions": {
                "class": "File",
                "path": "/var/lib/cwl/stgaa41a863-3253-4f23-92a6-2db7d5e3fec0/b1-analysis_blocks.bed"                                                 
            },
FarahZKhan commented 6 years ago

It runs fine without the --provenance flag now. So I am back to the initial issue i.e. 

Traceback (most recent call last):
  File "/home/ubuntu/cwlprov_testing/bin/cwltool", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/main.py", line 575, in main
    secret_store=runtimeContext.secret_store)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/main.py", line 222, in init_job_order
    basedir=file_uri(str(input_basedir) + "/"))
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/main.py", line 289, in printdeps
    {"$include", "$schemas", "location"}, loadref)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/process.py", line 921, in scandeps
    r.extend(scandeps(base, v, reffields, urlfields, loadref, urljoin=urljoin))
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/process.py", line 924, in scandeps
    r.extend(scandeps(base, d, reffields, urlfields, loadref, urljoin=urljoin))
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/process.py", line 927, in scandeps
    normalizeFilesDirs(r)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/pathmapper.py", line 82, in normalizeFilesDirs
    visit_class(job, ("File", "Directory"), addLocation)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/utils.py", line 230, in visit_class
    visit_class(rec[d], cls, op)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/utils.py", line 233, in visit_class
    visit_class(d, cls, op)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/utils.py", line 228, in visit_class
    op(rec)
  File "/home/ubuntu/cwlprov_testing/lib/python3.6/site-packages/cwltool/pathmapper.py", line 58, in addLocation
    raise validate.ValidationException("Anonymous file object must have 'contents' and 'basename' fields.")

EDIT: The input json file has "path" for each file object whereas somewhere in scapdeps or printdeps the code is looking for "location". I am not entirely sure about the base of the issue but if I replace all path: with location:, this particular error is resolved when running with --provenance: 

Anonymous file object must have 'contents' and 'basename' fields.