Open mr-c opened 3 years ago
The fix and/or its test seems flaky, so I reverted it in https://github.com/common-workflow-language/cwltool/pull/1483
Hi, any chance this can be fixed in the future? We have a CommandLineTool to move all intermediary files/folders to the destination path using bash script at Runtime:
requirements:
InlineJavascriptRequirement: {}
ShellCommandRequirement: {}
InitialWorkDirRequirement:
listing:
- entryname: mv.sh
entry: |-
shift;
mkdir $(inputs.destination);
# Move each file individually, ignoring non-existing files
for file in $@; do
if [ -e "$file" ]; then
mv -n "$file" "$(inputs.destination)"
fi
ls $(inputs.destination)
done
inputs:
files:
type: File[]?
inputBinding:
position: 2
folders:
type: Directory[]?
inputBinding:
position: 3
destination:
type: string
inputBinding:
position: 1
baseCommand: [bash, -x, mv.sh]
outputs:
results:
type: Directory
outputBinding:
glob: $(inputs.destination)
The above module does not work most of the time, and the output directory has only broken symlinks pointing to the file in tmpdir instead of real generated files, and these tmpdir does not exist. Sometimes the cwltool runs finally with a success, and from the log the mv process looks fine without error, but still in the outputs folder (destination) we only have broken symlinks. Typical error such as:
ERROR Unhandled error, try again with --debug for more information:
[Errno 2] No such file or directory: '/data/home/username/tmp/531ybmnk/aaaaaaa_illuminaQC_illumina_filtered_bbduk-summary.txt'
FYI, It is a remake of an older cwl script for the same purpose, we did it to minimize use of JS Expression as suggested in workflow description, and the old one use Javascript Expression:
requirements:
- class: InlineJavascriptRequirement
inputs:
files:
type: File[]?
folders:
type: Directory[]?
destination:
type: string
expression: |
${
var array = []
if (inputs.files != null) {
array = array.concat(inputs.files)
}
if (inputs.folders != null) {
array = array.concat(inputs.folders)
}
var r = {
'results':
{ "class": "Directory",
"basename": inputs.destination,
"listing": array
}
};
return r;
}
outputs:
results:
type: Directory
Thanks!
Short term workaround: use
--copy-outputs
but that leaves intermediate files laying around afterwards.Example error: