DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
893 stars 241 forks source link

How to use cwltool:overrides #2167

Open ionox0 opened 6 years ago

ionox0 commented 6 years ago

I've tried several methods of using cwltool:overrides in my inputs file that is supplied to Toil, but found an error for most methods:

Traceback (most recent call last):
  File "/Users/johnsoni/Desktop/virtualenv/msk/bin/toil-cwl-runner", line 11, in <module>
    sys.exit(main())
  File "/Users/johnsoni/Desktop/virtualenv/msk/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 962, in main
    options, sys.stdin, None, [], options.job_order)
  File "/Users/johnsoni/Desktop/virtualenv/msk/lib/python2.7/site-packages/cwltool/main.py", line 141, in load_job_order
    overrides.extend(resolve_overrides(job_order_object, file_uri(job_order_file), tool_file_uri))
  File "/Users/johnsoni/Desktop/virtualenv/msk/lib/python2.7/site-packages/cwltool/load_tool.py", line 366, in resolve_overrides
    ret, _ = ovloader.resolve_all(ov, baseurl)
  File "/Users/johnsoni/Desktop/virtualenv/msk/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 867, in resolve_all
    val, base_url+subscope, file_base=file_base, checklinks=False)
TypeError: can only concatenate list (not "str") to list

Tried methods:

cwltool:overrides:
  file:///Users/johnsoni/Desktop/code/Innovation-Pipeline/cwl_tools/marianas/ProcessLoopUMIFastq.cwl:
    requirements:
    - class: ResourceRequirement
      ramMin: 5000
      coresMin: 1
cwltool:overrides:
  ProcessLoopUMIFastq.cwl:
    requirements:
    - class: ResourceRequirement
      ramMin: 5000
      coresMin: 1
cwltool:overrides:
  "standard_pipeline.cwl":
    requirements:
    - class: ResourceRequirement
      ramMin: 5000
      coresMin: 1

Are there any suggestions for how to provide this as an input in the inputs.yaml file? Thanks!

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-246

DailyDreaming commented 4 years ago

@mr-c Is this functionality covered in the current cwl integration tests?

mr-c commented 4 years ago

It is not, as they are an extension to the CWL reference runner and not part of the standard (yet!)

It would be good to have this working in toil-cwl-runner