tetron
commented
7 years ago Some ideas:
The code that is responsible for evaluating js expressions is in cwltool/sandboxjs.py
There was a problem in the past with a conflicting program called node installed on some systems which wasn't Node.js. It is worth checking to see if there are node or nodejs executables in the path that do something unexpected.
If neither node or nodejs are present, it will pull the node:slim docker image and run that. The time spent pulling is not included in the timeout. However there was a bug (which I thought we fixed, but maybe not) where failure to pull would cause it to try to pull again in the "docker run" invocation, which would result in a mysterious time out.
The nodejs Popen object is stored in a thread-local variable so that the interpreter process can be reused to evaluate multiple expressions, but I don't know if that interacts with any forking or code pickling done by Toil.
Cwltool and Docker may be printing some stuff out to standard output / standard error that might give some clues, but I'm not sure if toil's logging is capturing it.
DailyDreaming
Figured there should still be an issue for this.
Running bcbio (from the GA4GH-DREAM challenge) on AWS gives an unknown Expression evaluation error in cwltool's expression.py (full output attached). It seems to get pretty far down the rabbit hole before the error shows up (about 2 hours and 45 minutes in). Will try running with a larger instance tomorrow (currently running a t2.2xlarge) to see if it either (hopefully) fixes some resource allocation problem, or runs faster.
bc_output.txt
┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-195