search

common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
332 stars 230 forks source link

Secondary Files required #1151

Open garyluu opened 5 years ago

garyluu commented 5 years ago

Expected Behavior

cwl v1.0 file with json that was able to run before with cwltool 1.0.20181217162649 should be able to be run with cwltool 1.0.20190621234233 . The secondary files are missing but still ran sucessfully before

Actual Behavior

ERROR Workflow error, try again with --debug for more information:
Missing required secondary file 'gdc-viral.dict.fai' from file object: {
    "basename": "gdc-viral.dict", 
    "nameroot": "gdc-viral", 
    "nameext": ".dict", 
    "location": "file:///home/gluu/dockstore/dockstore-client/src/test/resources/biodata/viral/gdc-viral.dict", 
    "secondaryFiles": [], 
    "class": "File", 
    "size": 0
}

Workflow Code

Workflow: https://github.com/dockstore/dockstore/blob/develop/dockstore-client/src/test/resources/directory/1st-workflow-no-secondary-in-workflow.cwl JSON: https://github.com/dockstore/dockstore/blob/develop/dockstore-client/src/test/resources/directory/1st-workflow-job.json

Full Traceback

INFO /home/gluu/dockstore/venv/bin/cwltool 1.0.20190621234233
INFO Resolved '/home/gluu/dockstore/dockstore-client/src/test/resources/directory/1st-workflow-no-secondary-in-workflow.cwl' to 'file:///home/gluu/dockstore/dockstore-client/src/test/resources/directory/1st-workflow-no-secondary-in-workflow.cwl'
ERROR Workflow error:
Missing required secondary file 'gdc-viral.dict.fai' from file object: {
    "basename": "gdc-viral.dict", 
    "nameroot": "gdc-viral", 
    "nameext": ".dict", 
    "location": "file:///home/gluu/dockstore/dockstore-client/src/test/resources/biodata/viral/gdc-viral.dict", 
    "secondaryFiles": [], 
    "class": "File", 
    "size": 0
}
Traceback (most recent call last):
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/main.py", line 823, in main
    logger=_logger)
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/executors.py", line 44, in __call__
    return self.execute(*args, **kwargs)
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/executors.py", line 103, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/executors.py", line 165, in run_jobs
    for job in jobiter:
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/workflow.py", line 578, in job
    builder = self._init_job(job_order, runtimeContext)
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/process.py", line 719, in _init_job
    discover_secondaryFiles=getdefault(runtime_context.toplevel, False)))
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/builder.py", line 277, in bind_input
    bindings.extend(self.bind_input(f, datum[f["name"]], lead_pos=lead_pos, tail_pos=f["name"], discover_secondaryFiles=discover_secondaryFiles))
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/builder.py", line 269, in bind_input
    bindings.extend(self.bind_input(st, datum, lead_pos=lead_pos, tail_pos=tail_pos, discover_secondaryFiles=discover_secondaryFiles))
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/builder.py", line 295, in bind_input
    self.bind_input(itemschema, item, lead_pos=n, tail_pos=tail_pos, discover_secondaryFiles=discover_secondaryFiles))
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/builder.py", line 269, in bind_input
    bindings.extend(self.bind_input(st, datum, lead_pos=lead_pos, tail_pos=tail_pos, discover_secondaryFiles=discover_secondaryFiles))
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/builder.py", line 295, in bind_input
    self.bind_input(itemschema, item, lead_pos=n, tail_pos=tail_pos, discover_secondaryFiles=discover_secondaryFiles))
  File "/home/gluu/dockstore/venv/local/lib/python2.7/site-packages/cwltool/builder.py", line 343, in bind_input
    sfname, json_dumps(datum, indent=4)))
WorkflowException: Missing required secondary file 'gdc-viral.dict.fai' from file object: {
    "basename": "gdc-viral.dict", 
    "nameroot": "gdc-viral", 
    "nameext": ".dict", 
    "location": "file:///home/gluu/dockstore/dockstore-client/src/test/resources/biodata/viral/gdc-viral.dict", 
    "secondaryFiles": [], 
    "class": "File", 
    "size": 0
}

Your Environment

mr-c commented 5 years ago

Hello @garyluu. i can confirm the change in behaviour.

While investigating this I found some unrelated bugs, so thank you! https://github.com/common-workflow-language/schema_salad/pull/265 https://github.com/common-workflow-language/cwltool/pull/1152