search

common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
335 stars 231 forks source link

CWL throws JavascriptException: error #955

Closed ranijames closed 6 years ago

ranijames commented 6 years ago

Dear All, I am running my CWL workflow on python 2.7 environment. I am trying to run couple python scripts, on BAM and tsv files. While I try to run my script. It is throwing following error. Following is the code. And I have all files suppiled in yml format

Workflow Code

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow

doc: "Workflow Components: increment -> Spladder -> PeptidePackage"

inputs:
 incremnet_tsv:
  type: File[]
 spladder_gtf:
  type: File 
 spladder_bams:
  type: File[]
 spladder_outDir:
  type: string
 spladder_phase2:
  type: string
 donor:
  type: string
 peptide_outDir:
  type: string
 reference:
  type: File

outputs:
 increment_out:
  type: File
  outputSource: increment_count/increment_out
 spladder_out1:
  type: Directory
  outputSource: spladder/spladder_out_dir1
 spladder_out2:
  type: Directory
  outputSource: spladder/spladder_out_dir2
 peptide_out:
  type: Directory
  outputSource: peptide/peptideOutDir

steps:
 increment_count:
  run: count_incrementally.cwl
  in:
   increment_out: increment_out
  out: [increment_out]
 spladder:
  run: spladder.cwl
  in:
   spladder_gtf: spladder_gtf
   spladder_bams: spladder_bams
   spladder_outDir: spladder_outDir
   spladder_phase2: spladder_phase2
  out: [spladder_out_dir1, spladder_out_dir2, spladderFile]
 peptide:
  run: peptide_package.cwl
  in:
   donor: donor
   peptide_outDir: peptide_outDir
   spliceFile: spladder/spladderFile
   annotation: spladder_gtf
   reference: reference 
  out: [peptideOutDir]

Full Traceback

Validation failed at
Traceback (most recent call last):
  File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/workflow.py", line 531, in __init__
    loadingContext.prov_obj))
  File "/cluster/home/aalva/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/workflow.py", line 689, in __init__
    raise validate.ValidationException("\n".join(validation_errors))
ValidationException: ../cwl/pipeline_part2.cwl:44:9: Workflow step output 'increment_out' does not correspond to
../cwl/count_incrementally.cwl:17:1:   tool output (expected 'incre_output')
I'm sorry, I couldn't load this CWL file.
The error was: 
Traceback (most recent call last):
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/main.py", line 673, in main
    metadata, uri, loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/load_tool.py", line 346, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/workflow.py", line 54, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/workflow.py", line 531, in __init__
    loadingContext.prov_obj))
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/workflow.py", line 619, in __init__
    toolpath_object["run"], loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/load_tool.py", line 378, in load_tool
    loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/load_tool.py", line 346, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/workflow.py", line 50, in default_make_tool
    return command_line_tool.CommandLineTool(toolpath_object, loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/command_line_tool.py", line 229, in __init__
    super(CommandLineTool, self).__init__(toolpath_object, loadingContext)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/process.py", line 595, in __init__
    validate_js_expressions(cast(CommentedMap, toolpath_object), self.doc_schema.names[toolpath_object["class"]], validate_js_options)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/validate_js.py", line 193, in validate_js_expressions
    expression_errors, _ = jshint_js(code_fragment_js, js_globals, jshint_options)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/validate_js.py", line 118, in jshint_js
    context=jshint_functions_text
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/sandboxjs.py", line 161, in exec_js_process
    new_proc = new_js_proc(js_engine_code, force_docker_pull=force_docker_pull)
  File "/cluster/home/user/software/anaconda/envs/py2/lib/python2.7/site-packages/cwltool/sandboxjs.py", line 110, in new_js_proc
    u"docker run node:slim".format(u", ".join(trynodes)))
JavascriptException: cwltool requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it.  Tried nodejs, node, docker run node:slim

Your Environment

biokcb commented 6 years ago

It looks like you just need to install node.js on your computer. Or docker so it can pull an image of it to run your CWL workflow. Try installing one of those and re-running.

edit: just noticed you're probably on a cluster, if it's a module based system you might just need to do a module load of node, otherwise ask the sysadmins to install it

ranijames commented 6 years ago

Thanks for your prompt reply. I am running CWL on cluster, where I cannot be super user for node.js

biokcb commented 6 years ago
  1. Ask the people that run the cluster if they can get node.js working for you
  2. If not that, then try removing the javascript expressions in your script, might be a tedious work around, I don't see where in your workflow it is used so I can't say for sure. (check the scripts that contain InlineJavascriptRequirement for the expressions)
ranijames commented 6 years ago

The javascript expressions are required for input for steps mentioned. I used that in my previous CWL workflow and it worked without complaining.

biokcb commented 6 years ago

Was the environment the same between runs? Seems odd that it would evaluate the JS Expression in one run but not the other. Perhaps the first run had some node/node.js module loaded?

ranijames commented 6 years ago

As far I understood DOCKer files are for each workflow separately, and I used all in-house bulit tools to run under my CWL workflow

mr-c commented 6 years ago

Hello @ranijames ,

This doesn't look like a primarily Javascript error, but an issue with how your workflow connects to the CWL tool descriptions.

Specifically look at

 increment_count:
  run: count_incrementally.cwl
  in:
   increment_out: increment_out
  out: [increment_out]

ValidationException: ../cwl/pipeline_part2.cwl:44:9: Workflow step output 'increment_out' does not correspond to ../cwl/count_incrementally.cwl:17:1: tool output (expected 'incre_output')

Here the count_incrementally.cwl tool doesn't have an output named increment_out, but it does have one named incre_output.

There may be an additional issue with a CWL parameter reference or expression in your tool descriptions, but without a link to your code I can't diagnose further.

ranijames commented 6 years ago

Hello Michael, Thanks for the comment and reply. My count_increment.cwl scripts looks like the following.

cwlVersion: v1.0
class: CommandLineTool
doc: "Script to incrementally count expression"

baseCommand: [python, /collect_counts_incrementally.py]

inputs:
 increment_out:
  type: string
  inputBinding:
   position: 1
 incremnet_tsv:
  type: File[]?
  inputBinding:
   position: 2

outputs:
 incre_output:
  type: File
  outputBinding:
   glob: $(inputs.increment_out)
mr-c commented 6 years ago

@ranijames You are welcome. Please try updating your step definition to match the actual output name:

 increment_count:
  run: count_incrementally.cwl
  in:
   increment_out: increment_out
  out: [incre_output]
ranijames commented 6 years ago

Hello Micheal, thanks for your suggestion and time, I changed the typos and re-ran my workflow. The error is still there.,

Resolved '/cluster/home/aalva/Projects/PHRT-Immuno/scripts/cwl/pipeline_part2.cwl' to 'file:///cluster/home/aalva/Projects/PHRT-Immuno/scripts/cwl/pipeline_part2.cwl'
I'm sorry, I couldn't load this CWL file.
The error was: 
Traceback (most recent call last):
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/main.py", line 673, in main
    metadata, uri, loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/load_tool.py", line 346, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/workflow.py", line 54, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/workflow.py", line 531, in __init__
    loadingContext.prov_obj))
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/workflow.py", line 619, in __init__
    toolpath_object["run"], loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/load_tool.py", line 378, in load_tool
    loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/load_tool.py", line 346, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/workflow.py", line 50, in default_make_tool
    return command_line_tool.CommandLineTool(toolpath_object, loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 229, in __init__
    super(CommandLineTool, self).__init__(toolpath_object, loadingContext)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/process.py", line 588, in __init__
    validate_js_expressions(cast(CommentedMap, toolpath_object), self.doc_schema.names[toolpath_object["class"]], validate_js_options)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/validate_js.py", line 193, in validate_js_expressions
    expression_errors, _ = jshint_js(code_fragment_js, js_globals, jshint_options)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/validate_js.py", line 118, in jshint_js
    context=jshint_functions_text
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/sandboxjs.py", line 161, in exec_js_process
    new_proc = new_js_proc(js_engine_code, force_docker_pull=force_docker_pull)
  File "/cluster/home/aalva/software/anaconda/lib/python3.6/site-packages/cwltool/sandboxjs.py", line 110, in new_js_proc
    u"docker run node:slim".format(u", ".join(trynodes)))
cwltool.sandboxjs.JavascriptException: cwltool requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it.  Tried nodejs, node, docker run node:slim
ranijames commented 6 years ago

Hello All,

I have a side question: How to source activate a python environment before a CWL step which needs a particular python version to run it?

ghost commented 6 years ago

I have a side question: How to source activate a python environment before a CWL step which needs a particular python version to run it?

Presumably you have this issue because you have not dockerized the tool and it is running bare on your local environment. In that case, you can have . activate ... as your first command, or create a bash script, and has that as your first command.

ranijames commented 6 years ago

Ok, Thanks for the comment. It worked when I ran it by calling in the script in python 2.7