"entry" content interpretation

[tatahin]

Expected Behavior

I tried to run CWL-file with bash-script inclusion, which makes new files.

Actual Behavior

It failed with error:

Workflow error, try again with --debug for more information:
Expression evaluation error:
Syntax error in parameter reference '{6}'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.

Part of Workflow Code

baseCommand: ["/bin/bash", "markduplicates_helper.sh"]
requirements:
    - class: ResourceRequirement
      coresMin: 8
      ramMin: 40000
    - class: DockerRequirement
      dockerPull: "mgibio/mark_duplicates-cwl:1.0.1"
    - class: InitialWorkDirRequirement
      listing:
      - entryname: 'markduplicates_helper.sh'
        entry: |
            set -o pipefail
            set -o errexit

            declare MD_BARCODE_TAG
            if [ ! -z "${6}" ]; then
              MD_BARCODE_TAG="BARCODE_TAG=${6}"
            /usr/bin/java -Xmx16g -jar /opt/picard/picard.jar MarkDuplicates I=$1 O=/dev/stdout ASSUME_SORT_ORDER=$5 METRICS_FILE=$4 QUIET=true COMPRESSION_LEVEL=0 VALIDATION_STRINGENCY=LENIENT "${MD_BARCODE_TAG}" > /usr/bin/sambamba sort -t $2 -m 18G -o $3 /dev/stdin
            else
              /usr/bin/java -Xmx16g -jar /opt/picard/picard.jar MarkDuplicates I=$1 O=/dev/stdout ASSUME_SORT_ORDER=$5 METRICS_FILE=$4 QUIET=true COMPRESSION_LEVEL=0 VALIDATION_STRINGENCY=LENIENT > /usr/bin/sambamba sort -t $2 -m 18G -o $3 /dev/stdin
            fi

Traceback

I've already added issue in The McDonnell Genome Institute repository. That's why authors stopped using ${VAR} syntax and tool started to work. But I just wanted to make sure that this code interpretation by cwltool is anticipated? Why is "entry" content interpreted by cwltool instead of separate interpretation?

Your Environment

• cwltool version: 1.0.20190228155703

[mr-c]

Hello @tatahin, and thanks for this issue. I think we should improve the error message in this case.

This field allows for CWL references and/or CWL expressions:
https://www.commonwl.org/v1.0/CommandLineTool.html#Dirent

Therefore one must escape all uses of $ in this field to avoid interpretation as a CWL reference or CWL expression

[serge2016]

Hello, @mr-c! I think, that it would be great to explain (and maybe to give an example), how to pass the text of a bash script through entry.

Maybe CWL should have some requirement or option to pass the value of entry as is and on other case to interpret the expressions...?

[tatahin]

@mr-c, thanks for your answer.

[ghost]

@serge2016

common-workflow-language/common-workflow-language#852 https://github.com/common-workflow-language/common-workflow-language/issues/850

(There are some interesting code snippets in the comments) https://github.com/common-workflow-language/common-workflow-language/issues/850#issuecomment-483738499

https://www.commonwl.org/user_guide/14-runtime/index.html

In general I agree this aspect of CWL could be vastly improved. It's way too difficult to do this simple task of creating a CLT out of a bash script.

It's trivial if you have the bash script as part of the docker image, but otherwise it's hell.

Also, here is a page under construction under rabix.io. It is not "officially" presentable yet:

http://rabix.io/cwl-patterns.html