Open jeremiahsavage opened 7 years ago
Bunny is failing because there are no positions in the inputs section. Because of that, Bunny is building the following command line:
aws /Users/janko/Desktop/cwl_aws_cp/aws/endpoint.json /Users/janko/Desktop/cwl_aws_cp/aws/signpost.json s3 cp --endpoint-url http://s3-us-east-2.amazonaws.com --profile us-east-2 --no-sign-request s3://readonly-test-bucket/empty.txt .
Positions are not well defined in the spec and we are working on that. For now, you can fix the tool by assigning explicit positions to the inputs.
One more thing. You need to use valueFrom property in order to put file contents to the command line.
Thanks for the feedback.
The reason the tool has valueFrom: null
for two of the input files is that I only want to place a select substring of those files on the commandline, using arguments
. I found if I loadContents: true
without valueFrom: null
, then cwltool will place the two file paths on the commandline, which is not my intention. The only way the tool can function is with arguments, as the input values are not known ahead of time.
I tried creating a new tool with one input having a position 0, but it does not succeed. Looking more closely at the bunny cmd
, it looks like bunny is placing the endpoint_json
and signpost_json
file paths on the commandline, while cwltool does not do this (due to valueFrom: null
)
bunny should not inject /home/jeremiah/cwl_aws_cp/aws/endpoint.json home/jeremiah/cwl_aws_cp/aws/signpost.json
:
cmd=[/bin/sh, -c, aws /home/jeremiah/cwl_aws_cp/aws/endpoint.json /home/jeremiah/cwl_aws_cp/aws/signpost.json s3 cp --endpoint-url http://s3-us-east-2.amazonaws.com --profile us-east-2 --no-sign-request s3://readonly-test-bucket/empty.txt .]
cwltool does not:
/bin/sh \
-c \
'aws' 's3' 'cp' '--endpoint-url' 'http://s3-us-east-2.amazonaws.com' '--profile' 'us-east-2' '--no-sign-request' 's3://readonly-test-bucket/empty.txt' '.'
this is with a tool that contains one input with a position:
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
requirements:
- class: DockerRequirement
dockerPull: quay.io/ncigdc/awscli:1
- class: EnvVarRequirement
envDef:
- envName: "AWS_CONFIG_FILE"
envValue: $(inputs.aws_config.path)
- envName: "AWS_SHARED_CREDENTIALS_FILE"
envValue: $(inputs.aws_shared_credentials.path)
- class: InlineJavascriptRequirement
- class: ShellCommandRequirement
inputs:
- id: aws
type: string
default: aws
inputBinding:
position: 0
- id: aws_config
type: File
- id: aws_shared_credentials
type: File
- id: endpoint_json
type: File
inputBinding:
loadContents: true
valueFrom: null
- id: signpost_json
type: File
inputBinding:
loadContents: true
valueFrom: null
outputs:
- id: output
type: File
outputBinding:
glob: |
${
var signpost_json = JSON.parse(inputs.signpost_json.contents);
var signpost_url = String(signpost_json.urls.slice(0));
var file_name = signpost_url.split('/').slice(-1)[0];
return file_name
}
arguments:
- valueFrom: "s3"
position: 1
shellQuote: true
- valueFrom: "cp"
position: 2
shellQuote: true
- valueFrom: |
${
var signpost_json = JSON.parse(inputs.signpost_json.contents);
var signpost_url = String(signpost_json.urls.slice(0));
var signpost_path = signpost_url.slice(5);
var signpost_array = signpost_path.split('/');
var signpost_root = signpost_array[0];
var profile = signpost_root;
var endpoint_json = JSON.parse(inputs.endpoint_json.contents);
var endpoint_url = String(endpoint_json[profile]);
return endpoint_url
}
prefix: --endpoint-url
position: 4
shellQuote: true
- valueFrom: |
${
var signpost_json = JSON.parse(inputs.signpost_json.contents);
var signpost_url = String(signpost_json.urls.slice(0));
var signpost_path = signpost_url.slice(5);
var signpost_array = signpost_path.split('/');
var signpost_root = signpost_array[0];
var profile = signpost_root;
return profile
}
prefix: --profile
position: 5
shellQuote: true
- valueFrom: "--no-sign-request"
position: 6
shellQuote: true
- valueFrom: |
${
var signpost_json = JSON.parse(inputs.signpost_json.contents);
var obj_path = [];
obj_path.push(signpost_json.urls[0]);
var signpost_path = obj_path[0].substring(5).split('/').slice(1).join('/');
var s3_url = "s3://" + signpost_path;
return s3_url
}
position: 98
shellQuote: true
- valueFrom: .
position: 99
shellQuote: true
baseCommand: []
By the spec, valueFrom can be string or an expression. I think that that is a bug in cwltool. One more thing, if the valueFrom is null that is more or less the same thing as if the valueFrom is missing. The correct way would be the following:
valueFrom : $(null)
With that change and positions, Bunny produced the following result:
{
"output" : {
"basename" : "empty.txt",
"checksum" : "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
"class" : "File",
"dirname" : "/Users/janko/Desktop/cwl_aws_cp/fb4a53b4-6ba0-482a-a83e-b1ec3b9f5c48/root",
"location" : "/Users/janko/Desktop/cwl_aws_cp/fb4a53b4-6ba0-482a-a83e-b1ec3b9f5c48/root/empty.txt",
"nameext" : "txt",
"nameroot" : "empty",
"path" : "/Users/janko/Desktop/cwl_aws_cp/fb4a53b4-6ba0-482a-a83e-b1ec3b9f5c48/root/empty.txt",
"secondaryFiles" : [ ],
"size" : 0
}
}
The valueFrom: $(null)
fix works for me on bunny.
However, with cwltool, I get an error:
Workflow error, try again with --debug for more information:
:1:1: Expression evaluation error:
null'null'
Traceback (most recent call last):
File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/main.py", line 768, in main
**vars(args))
File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/main.py", line 235, in single_job_executor
for r in jobiter:
File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/draft2tool.py", line 378, in job
arg = builder.generate_arg(b)
File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/cwltool/builder.py", line 175, in generate_arg
value = self.do_eval(binding["valueFrom"], context=value)
File "/home/jeremiah/.virtualenvs/cwl/lib/python2.7/site-packages/schema_salad/sourceline.py", line 150, in __exit__
raise self.makeError(unicode(exc_value))
WorkflowException: :1:1: Expression evaluation error:
:1:1: null'null'
I'm not sure where in the spec this is specified. Should I file a bug against cwltool? Edit: Never mind. I see issue 373 created above.
Hi @jeremiahsavage , you can also set valueFrom : "", i.e. add an empty string to the command line. This should work everywhere.
@bogdang989 That's only true if there is no prefix. Both rabix and cwltool will run echo -p
for example, instead of just echo
.
@jeremiahsavage I thought for a moment that you don't have a InlineJavascriptRequirement declared, but since you do, I believe this is a bug in cwltool.
@tetron @mr-c can you share your thoughts about this issue? The specification says that valueFrom can be string | Expression, so I believe that the right way of specifying null in valueFrom is through expression as @simonovic86 suggested.
@sivkovic yes, failing to handle $(null) is a bug in cwltool. I have a pending bugfix on cwltool https://github.com/common-workflow-language/cwltool/pull/434 and added test cases to the conformance suite: https://github.com/common-workflow-language/common-workflow-language/pull/460
@tetron thanks for fast response.
git clone https://github.com/jeremiahsavage/cwl_aws_cp.git
with cwltool:
with the develop branch of bunny: