search

Sage-Bionetworks / SynapseWorkflowHook

Code for linking a workflow engine to a Synapse evaluation queue
Apache License 2.0
4 stars 1 forks source link

No TOIL_CLI_OPTIONS causes incorrect cwl-toil run command #17

Closed thomasyu888 closed 6 years ago

thomasyu888 commented 6 years ago

toil cmd: [toil-cwl-runner, , --workDir, /home/ubuntu/temp_dir2/b53d521a-0310-43cd-9c1f-fb88764b72b3, --noLinkImports, ChallengeWorkflowTemplates-master/scoringHarness_workflow.cwl, /home/ubuntu/temp_dir2/b53d521a-0310-43cd-9c1f-fb88764b72b3/TMP7978463776103023382.yaml]

thomasyu888 commented 6 years ago

Notice the space in-between toil-cwl-runner, , --workDir

brucehoff commented 6 years ago

Looks like this causes a run time error:

STDERR: 2018-10-09T18:19:11.318753400Z INFO:cwltool:Resolved '' to 'file:///Users/bhoff/temp/ad42da82-c0f4-4916-8ab7-777c22ef7ba2'
STDERR: 2018-10-09T18:19:11.322205800Z Traceback (most recent call last):
STDERR: 2018-10-09T18:19:11.322231500Z   File "/usr/local/bin/toil-cwl-runner", line 11, in <module>
STDERR: 2018-10-09T18:19:11.322239400Z     sys.exit(main())
STDERR: 2018-10-09T18:19:11.322246100Z   File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1140, in main
STDERR: 2018-10-09T18:19:11.322426700Z     loading_context.fetcher_constructor)
STDERR: 2018-10-09T18:19:11.322442900Z   File "/usr/local/lib/python2.7/site-packages/cwltool/load_tool.py", line 113, in fetch_document
STDERR: 2018-10-09T18:19:11.322625600Z     workflowobj = document_loader.fetch(fileuri)
STDERR: 2018-10-09T18:19:11.322639800Z   File "/usr/local/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 927, in fetch
STDERR: 2018-10-09T18:19:11.324051500Z     text = self.fetch_text(url)
STDERR: 2018-10-09T18:19:11.324067300Z   File "/usr/local/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 154, in fetch_text
STDERR: 2018-10-09T18:19:11.324107700Z     raise RuntimeError(Text(err))
STDERR: 2018-10-09T18:19:11.324186700Z RuntimeError: [Errno 21] Is a directory: '/Users/bhoff/temp/ad42da82-c0f4-4916-8ab7-777c22ef7ba2'
brucehoff commented 6 years ago

I fixed the blank command line argument: https://github.com/Sage-Bionetworks/SynapseWorkflowHook/commit/fdccc423d17705c61be376a9dcdaec2baae85ad2

However you will now probably find that workflows fail because you have not passed various toil options. (See the stack trace at bottom.) I have found, through trial and error, that these settings work:

TOIL_CLI_OPTIONS=--defaultMemory 100M --retryCount 0 --defaultDisk 1000000

Without any options, I get:

STDERR: 2018-10-09T18:28:34.380140900Z INFO:cwltool:Resolved 'SynapseWorkflowExample-master/workflow-entrypoint.cwl' to 'file:///Users/bhoff/temp/8e230d43-46f4-4cf9-8222-570581520ea9/SynapseWorkflowExample-master/workflow-entrypoint.cwl'
STDERR: 2018-10-09T18:28:35.994497700Z WARNING:cwltool:Workflow checker warning:
STDERR: 2018-10-09T18:28:35.994527500Z SynapseWorkflowExample-master/workflow-entrypoint.cwl:48:9: Source 'message' of type ["null",
STDERR: 2018-10-09T18:28:35.994537200Z                                                             "string"] may be incompatible
STDERR: 2018-10-09T18:28:35.994546000Z SynapseWorkflowExample-master/workflow-entrypoint.cwl:53:9:   with sink 'message' of type "string"
STDERR: 2018-10-09T18:28:36.242982500Z WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (6).
STDERR: 2018-10-09T18:28:36.243007900Z WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (2095468544).
STDERR: 2018-10-09T18:28:36.244072200Z WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (245185695744).
STDERR: 2018-10-09T18:28:36.271311400Z INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
STDERR: 2018-10-09T18:28:37.289966100Z Traceback (most recent call last):
STDERR: 2018-10-09T18:28:37.289996100Z   File "/usr/local/bin/toil-cwl-runner", line 11, in <module>
STDERR: 2018-10-09T18:28:37.290005400Z     sys.exit(main())
STDERR: 2018-10-09T18:28:37.290014700Z   File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1220, in main
STDERR: 2018-10-09T18:28:37.290024800Z     outobj = toil.start(wf1)
STDERR: 2018-10-09T18:28:37.290033800Z   File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 784, in start
STDERR: 2018-10-09T18:28:37.290065500Z     return self._runMainLoop(rootJobGraph)
STDERR: 2018-10-09T18:28:37.290080300Z   File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 1059, in _runMainLoop
STDERR: 2018-10-09T18:28:37.290183800Z     jobCache=self._jobCache).run()
STDERR: 2018-10-09T18:28:37.290193900Z   File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 208, in run
STDERR: 2018-10-09T18:28:37.290756800Z     self.innerLoop()
STDERR: 2018-10-09T18:28:37.290774000Z   File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 515, in innerLoop
STDERR: 2018-10-09T18:28:37.290999100Z     self._processReadyJobs()
STDERR: 2018-10-09T18:28:37.291014600Z   File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 439, in _processReadyJobs
STDERR: 2018-10-09T18:28:37.291078100Z     self._processReadyJob(jobGraph, resultStatus)
STDERR: 2018-10-09T18:28:37.291096200Z   File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 389, in _processReadyJob
STDERR: 2018-10-09T18:28:37.291104800Z     self.issueJob(JobNode.fromJobGraph(jobGraph))
STDERR: 2018-10-09T18:28:37.291112200Z   File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 587, in issueJob
STDERR: 2018-10-09T18:28:37.291190300Z     jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
STDERR: 2018-10-09T18:28:37.291202500Z   File "/usr/local/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 234, in issueBatchJob
STDERR: 2018-10-09T18:28:37.291483000Z     'with.'.format(jobNode.jobName, jobNode.memory, self.maxMemory))
STDERR: 2018-10-09T18:28:37.291496200Z AssertionError: The job CWLWorkflow is requesting 2147483648 bytes of memory, more than the maximum of 2095468544 this batch system was configured with.
thomasyu888 commented 6 years ago

I feel like then these parameters shouldn't be optional then right? Or they are highly recommended because of errors?

brucehoff commented 6 years ago

I feel like then these parameters shouldn't be optional then right?

Maybe, but we are getting into how Toil works (or how it works on a specific platform, in this case Docker running on MacOS). I don't think it's the job of the Workflow Hook to figure out what options are needed for the chosen platform. I do include the options I have found to work in the README so you can use them.