DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
900 stars 240 forks source link

Toil's CWL implementation doesn't support `**` exponentiation in CWL experssions #4065

Closed adamnovak closed 2 years ago

adamnovak commented 2 years ago

I tried to use ** in a CWL tool definition, alongside InlineJavascriptRequirement.

When I tried to run it with Toil, I got:

    [2022-03-16T18:09:40+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
    [2022-03-16T18:09:40+0000] [MainThread] [I] [toil] Running Toil version 5.7.0a1-9ac4159f00d1150269ae041138d2db3fd4844794 on host ip-10-0-190-15.us-west-2.compute.internal.
    [2022-03-16T18:09:40+0000] [MainThread] [I] [toil.worker] Working on job 'CWLJob' nontrivial.cwl.7.sort-file.cwl fc505ed6-2fda-4255-9847-91085e24d2df v4
    [2022-03-16T18:09:41+0000] [MainThread] [I] [toil.worker] Loaded body Job('CWLJob' nontrivial.cwl.7.sort-file.cwl fc505ed6-2fda-4255-9847-91085e24d2df v4) from description 'CWLJob' nontrivial.cwl.7.sort-file.cwl fc505ed6-2fda-4255-9847-91085e24d2df v4
    [2022-03-16T18:09:41+0000] [MainThread] [E] [cwltool] Expecting value: line 1 column 1 (char 0)
    script was:
    01 "use strict";
    02 var inputs = {
    03     "input_file": {
    04         "location": "toilfile:359133:0:2df7fc26-b7e2-4bfb-94e9-23196027503c",
    05         "basename": "output.txt",
    06         "nameroot": "output",
    07         "nameext": ".txt",
    08         "class": "File",
    09         "checksum": "sha1$5e7b8ec1a00d1269e4d3a44217b64d55bb492ad2",
    10         "size": 359133,
    11         "http://commonwl.org/cwltool#generation": 0
    12     }
    13 };
    14 var self = null;
    15 var runtime = {
    16     "tmpdir": "/var/lib/toil/b9615395ef695cb1adfb158e0bafb57a/5264/c351/tmp9j2nss59ivifvisv",
    17     "outdir": "/var/lib/toil/b9615395ef695cb1adfb158e0bafb57a/5264/c351/tmptzu0g68x/tmp-outamch6nde"
    18 };
    19 (function(){return ((parseInt(Math.ceil(inputs.input_file.size/(2**20)))));})()
    stdout was: ''
    stderr was: 'evalmachine.<anonymous>:19
    (function(){return ((parseInt(Math.ceil(inputs.input_file.size/(2**20)))));})()
                                                                      ^

    SyntaxError: Unexpected token *
        at createScript (vm.js:56:10)
        at Object.runInNewContext (vm.js:93:10)
        at Socket.<anonymous> ([eval]:11:57)
        at emitOne (events.js:96:13)
        at Socket.emit (events.js:188:7)
        at readableAddChunk (_stream_readable.js:176:18)
        at Socket.Readable.push (_stream_readable.js:134:10)
        at Pipe.onread (net.js:559:20)'
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/dist-packages/cwltool/sandboxjs.py", line 384, in execjs
        return cast(CWLOutputType, json.loads(stdout))
      File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

It looks like whatever Node Toil is using to execute the Javascript doesn't know about the ** operator.

Javascript hasn't always had a ** operator, and the CWL spec is a bit unclear on what Javascript version exactly we are supposed to implement, but this works under cwltool for me.

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1152

mr-c commented 2 years ago

CWL v1.x uses ECMAscript 5.1 which does not have the ** operator https://www.commonwl.org/v1.2/CommandLineTool.html#Parameter_references