fail to execute with awscli

[jeremiahsavage]

git clone https://github.com/jeremiahsavage/cwl_aws_cp.git

with cwltool:

(cwl) [jeremiah@localhost ~]$ cwltool cwl_aws_cp/aws_s3_get.cwl cwl_aws_cp/aws_s3_get.yml 
/home/jeremiah/.virtualenvs/cwl/bin/cwltool 1.0.20170413194156
Resolved 'cwl_aws_cp/aws_s3_get.cwl' to 'file:///home/jeremiah/cwl_aws_cp/aws_s3_get.cwl'
[job aws_s3_get.cwl] /tmp/tmpXzacA9$ docker \
    run \
    -i \
    --volume=/home/jeremiah/cwl_aws_cp/aws/config:/var/lib/cwl/stg7baa7395-1376-41a8-bfdc-ac1e41765685/config:ro \
    --volume=/home/jeremiah/cwl_aws_cp/aws/credentials:/var/lib/cwl/stgb922b629-4142-439a-93ec-daa3f6bdc669/credentials:ro \
    --volume=/home/jeremiah/cwl_aws_cp/aws/endpoint.json:/var/lib/cwl/stg7fd14e9c-f5e4-4de1-867a-c70cc81fcf33/endpoint.json:ro \
    --volume=/home/jeremiah/cwl_aws_cp/aws/signpost.json:/var/lib/cwl/stg0a831e6c-66ce-443a-899f-984bfb2dfcba/signpost.json:ro \
    --volume=/tmp/tmpXzacA9:/var/spool/cwl:rw \
    --volume=/tmp/tmpp7NRlv:/tmp:rw \
    --workdir=/var/spool/cwl \
    --read-only=true \
    --user=1000 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/var/spool/cwl \
    --env=AWS_SHARED_CREDENTIALS_FILE=/var/lib/cwl/stgb922b629-4142-439a-93ec-daa3f6bdc669/credentials \
    --env=AWS_CONFIG_FILE=/var/lib/cwl/stg7baa7395-1376-41a8-bfdc-ac1e41765685/config \
    quay.io/ncigdc/awscli:1 \
    /bin/sh \
    -c \
    'aws' 's3' 'cp' '--endpoint-url' 'http://s3-us-east-2.amazonaws.com' '--profile' 'us-east-2' 's3://readonly-test-bucket/empty.txt' '.'
download: s3://readonly-test-bucket/empty.txt to ./empty.txt
[job aws_s3_get.cwl] completed success
{
    "output": {
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709", 
        "basename": "empty.txt", 
        "location": "file:///home/jeremiah/empty.txt", 
        "path": "/home/jeremiah/empty.txt", 
        "class": "File", 
        "size": 0
    }
}
Final process status is success

with the develop branch of bunny:

(cwl) [jeremiah@localhost ~]$ java -jar ~/code/bunny/rabix-backend-local/target/rabix-backend-local-1.0.0-rc4.jar cwl_aws_cp/aws_s3_get.cwl cwl_aws_cp/aws_s3_get.yml
[2017-04-19 15:51:13.197] [DEBUG] Config path: /home/jeremiah/code/bunny/rabix-backend-local/config
[2017-04-19 15:51:13.199] [DEBUG] Configuration directory found localy.
[2017-04-19 15:51:14.226] [DEBUG] DAGCache initialized with size=16
[2017-04-19 15:51:14.227] [DEBUG] ApplicationCache initialized with size=16
...
...
...
[2017-04-19 15:51:16.684] [INFO] Starting container with Id: 9561c54a2dd35936c3eaadb9e43c9e447ce1b99c376de3060b1482d9b5da0477
[2017-04-19 15:51:16.872] [INFO] Docker container 9561c54a2dd35936c3eaadb9e43c9e447ce1b99c376de3060b1482d9b5da0477 has started.
[2017-04-19 15:51:16.873] [DEBUG] Command STATUS found. Start execution.
[2017-04-19 15:51:16.873] [DEBUG] Command STATUS is repeatable. Delay and put it back to queue.
[2017-04-19 15:51:17.873] [DEBUG] Check status for e9b8c856-9587-4007-92c2-0755c5d11520 command line tool.
[2017-04-19 15:51:17.924] [DEBUG] postprocess(id=e9b8c856-9587-4007-92c2-0755c5d11520)
[2017-04-19 15:51:17.932] [INFO] Trying to find cached results in the directory /Users/janko/Desktop/Archive/varscan/.root.meta
[2017-04-19 15:51:17.932] [INFO] Cache directory doesn't exist. Directory /Users/janko/Desktop/Archive/varscan/.root.meta
[2017-04-19 15:51:17.934] [DEBUG] Saving standard error files for id=e9b8c856-9587-4007-92c2-0755c5d11520
[2017-04-19 15:51:18.104] [DEBUG] isSuccessful()
[2017-04-19 15:51:18.104] [DEBUG] getExitStatus()
[2017-04-19 15:51:18.120] [DEBUG] isSuccessful()
[2017-04-19 15:51:18.121] [DEBUG] getExitStatus()
[2017-04-19 15:51:18.136] [DEBUG] getExitStatus()
[2017-04-19 15:51:18.142] [ERROR] Job e9b8c856-9587-4007-92c2-0755c5d11520 failed with exit code 2.
[2017-04-19 15:51:18.142] [DEBUG] onJobFailed(jobId=e9b8c856-9587-4007-92c2-0755c5d11520)
[2017-04-19 15:51:18.144] [DEBUG] Update Job e9b8c856-9587-4007-92c2-0755c5d11520
[2017-04-19 15:51:18.144] [DEBUG] Root Job e9b8c856-9587-4007-92c2-0755c5d11520 goes to EventProcessor 0
[2017-04-19 15:51:18.144] [DEBUG] Root Job e9b8c856-9587-4007-92c2-0755c5d11520 goes to EventProcessor 0
[2017-04-19 15:51:18.145] [INFO] Job e9b8c856-9587-4007-92c2-0755c5d11520 failed with exit code 2.
[2017-04-19 15:51:18.145] [DEBUG] isSuccessful()
[2017-04-19 15:51:18.145] [DEBUG] getExitStatus()
[2017-04-19 15:51:18.159] [DEBUG] Command STATUS is last command. Stop thread.
[2017-04-19 15:51:18.159] [INFO] JobHandlerRunnable JobHandler-Thread-0 stopped.
[2017-04-19 15:51:18.159] [INFO] JobHandlerRunnable JobHandler-Thread-0 finished.
[2017-04-19 15:51:18.234] [DEBUG] DAGNode rootId=e9b8c856-9587-4007-92c2-0755c5d11520, id=root found in cache
[2017-04-19 15:51:18.234] [DEBUG] Cache size=1
[2017-04-19 15:51:18.240] [WARN] Root job e9b8c856-9587-4007-92c2-0755c5d11520 failed.
[2017-04-19 15:51:18.240] [DEBUG] onJobFailed(jobId=e9b8c856-9587-4007-92c2-0755c5d11520)
[2017-04-19 15:51:18.240] [DEBUG] Root Job e9b8c856-9587-4007-92c2-0755c5d11520 goes to EventProcessor 0
[2017-04-19 15:51:18.241] [DEBUG] org.rabix.engine.memory.impl.InMemoryJobRecordRepository@ee38559 flushed 1 item(s). Cache hits 1.
[2017-04-19 15:51:18.241] [DEBUG] org.rabix.engine.memory.impl.InMemoryVariableRecordRepository@32b45313 flushed 0 item(s). Cache hits 0.
[2017-04-19 15:51:18.374] [INFO] Failed to execute a Job

[simonovic86]

Bunny is failing because there are no positions in the inputs section. Because of that, Bunny is building the following command line:

aws /Users/janko/Desktop/cwl_aws_cp/aws/endpoint.json /Users/janko/Desktop/cwl_aws_cp/aws/signpost.json s3 cp --endpoint-url http://s3-us-east-2.amazonaws.com --profile us-east-2 --no-sign-request s3://readonly-test-bucket/empty.txt .

Positions are not well defined in the spec and we are working on that. For now, you can fix the tool by assigning explicit positions to the inputs.

One more thing. You need to use valueFrom property in order to put file contents to the command line.

[jeremiahsavage]

Thanks for the feedback.

The reason the tool has valueFrom: null for two of the input files is that I only want to place a select substring of those files on the commandline, using arguments. I found if I loadContents: true without valueFrom: null, then cwltool will place the two file paths on the commandline, which is not my intention. The only way the tool can function is with arguments, as the input values are not known ahead of time.

I tried creating a new tool with one input having a position 0, but it does not succeed. Looking more closely at the bunny cmd, it looks like bunny is placing the endpoint_json and signpost_json file paths on the commandline, while cwltool does not do this (due to valueFrom: null)

bunny should not inject /home/jeremiah/cwl_aws_cp/aws/endpoint.json home/jeremiah/cwl_aws_cp/aws/signpost.json:

cmd=[/bin/sh, -c, aws /home/jeremiah/cwl_aws_cp/aws/endpoint.json /home/jeremiah/cwl_aws_cp/aws/signpost.json s3 cp --endpoint-url http://s3-us-east-2.amazonaws.com --profile us-east-2 --no-sign-request s3://readonly-test-bucket/empty.txt .]

cwltool does not:

/bin/sh \
    -c \
    'aws' 's3' 'cp' '--endpoint-url' 'http://s3-us-east-2.amazonaws.com' '--profile' 'us-east-2' '--no-sign-request' 's3://readonly-test-bucket/empty.txt' '.'

this is with a tool that contains one input with a position:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0

class: CommandLineTool
requirements:
  - class: DockerRequirement
    dockerPull: quay.io/ncigdc/awscli:1
  - class: EnvVarRequirement
    envDef:
      - envName: "AWS_CONFIG_FILE"
        envValue: $(inputs.aws_config.path)
      - envName: "AWS_SHARED_CREDENTIALS_FILE"
        envValue: $(inputs.aws_shared_credentials.path)
  - class: InlineJavascriptRequirement
  - class: ShellCommandRequirement

inputs:
  - id: aws
    type: string
    default: aws
    inputBinding:
      position: 0

  - id: aws_config
    type: File

  - id: aws_shared_credentials
    type: File

  - id: endpoint_json
    type: File
    inputBinding:
      loadContents: true
      valueFrom: null

  - id: signpost_json
    type: File
    inputBinding:
      loadContents: true
      valueFrom: null

outputs:
  - id: output
    type: File
    outputBinding:
      glob: |
        ${
        var signpost_json = JSON.parse(inputs.signpost_json.contents);
        var signpost_url = String(signpost_json.urls.slice(0));
        var file_name = signpost_url.split('/').slice(-1)[0];
        return file_name
        }

arguments:
  - valueFrom: "s3"
    position: 1
    shellQuote: true

  - valueFrom: "cp"
    position: 2
    shellQuote: true

  - valueFrom: |
      ${
      var signpost_json = JSON.parse(inputs.signpost_json.contents);
      var signpost_url = String(signpost_json.urls.slice(0));
      var signpost_path = signpost_url.slice(5);
      var signpost_array = signpost_path.split('/');
      var signpost_root = signpost_array[0];
      var profile = signpost_root;
      var endpoint_json = JSON.parse(inputs.endpoint_json.contents);
      var endpoint_url = String(endpoint_json[profile]);
      return endpoint_url
      }
    prefix: --endpoint-url
    position: 4
    shellQuote: true

  - valueFrom: |
      ${
      var signpost_json = JSON.parse(inputs.signpost_json.contents);
      var signpost_url = String(signpost_json.urls.slice(0));
      var signpost_path = signpost_url.slice(5);
      var signpost_array = signpost_path.split('/');
      var signpost_root = signpost_array[0];
      var profile = signpost_root;
      return profile
      }
    prefix: --profile
    position: 5
    shellQuote: true

  - valueFrom: "--no-sign-request"
    position: 6
    shellQuote: true

  - valueFrom: |
      ${
      var signpost_json = JSON.parse(inputs.signpost_json.contents);
      var obj_path = [];
      obj_path.push(signpost_json.urls[0]);
      var signpost_path = obj_path[0].substring(5).split('/').slice(1).join('/');
      var s3_url = "s3://" + signpost_path;
      return s3_url
      }
    position: 98
    shellQuote: true

  - valueFrom: .
    position: 99
    shellQuote: true

baseCommand: []

[simonovic86]

By the spec, valueFrom can be string or an expression. I think that that is a bug in cwltool. One more thing, if the valueFrom is null that is more or less the same thing as if the valueFrom is missing. The correct way would be the following:

valueFrom : $(null)

With that change and positions, Bunny produced the following result:

{
  "output" : {
    "basename" : "empty.txt",
    "checksum" : "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
    "class" : "File",
    "dirname" : "/Users/janko/Desktop/cwl_aws_cp/fb4a53b4-6ba0-482a-a83e-b1ec3b9f5c48/root",
    "location" : "/Users/janko/Desktop/cwl_aws_cp/fb4a53b4-6ba0-482a-a83e-b1ec3b9f5c48/root/empty.txt",
    "nameext" : "txt",
    "nameroot" : "empty",
    "path" : "/Users/janko/Desktop/cwl_aws_cp/fb4a53b4-6ba0-482a-a83e-b1ec3b9f5c48/root/empty.txt",
    "secondaryFiles" : [ ],
    "size" : 0
  }
}

[jeremiahsavage]

The valueFrom: $(null) fix works for me on bunny.

However, with cwltool, I get an error:

Workflow error, try again with --debug for more information:
:1:1: Expression evaluation error:
      null'null'
Traceback (most recent call last):
  File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/main.py", line 768, in main
    **vars(args))
  File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/main.py", line 235, in single_job_executor
    for r in jobiter:
  File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/draft2tool.py", line 378, in job
    arg = builder.generate_arg(b)
  File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/builder.py", line 175, in generate_arg
    value = self.do_eval(binding["valueFrom"], context=value)
  File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/schema_salad/sourceline.py", line 150, in __exit__
    raise self.makeError(unicode(exc_value))
WorkflowException: :1:1: Expression evaluation error:
:1:1: null'null'

I'm not sure where in the spec this is specified. Should I file a bug against cwltool? Edit: Never mind. I see issue 373 created above.

[bogdang989]

Hi @jeremiahsavage , you can also set valueFrom : "", i.e. add an empty string to the command line. This should work everywhere.

[StarvingMarvin]

@bogdang989 That's only true if there is no prefix. Both rabix and cwltool will run echo -p for example, instead of just echo. @jeremiahsavage I thought for a moment that you don't have a InlineJavascriptRequirement declared, but since you do, I believe this is a bug in cwltool.

[sivkovic]

@tetron @mr-c can you share your thoughts about this issue? The specification says that valueFrom can be string | Expression, so I believe that the right way of specifying null in valueFrom is through expression as @simonovic86 suggested.

[tetron]

@sivkovic yes, failing to handle $(null) is a bug in cwltool. I have a pending bugfix on cwltool https://github.com/common-workflow-language/cwltool/pull/434 and added test cases to the conformance suite: https://github.com/common-workflow-language/common-workflow-language/pull/460

[sivkovic]

@tetron thanks for fast response.