Open stxue1 opened 3 months ago
@stxue1 Thanks, I think there are two slightly separate topics here:
File?
with a nonexistent path successfully evaluating to None
/null
is scoped to the task output section and (IMO) not applicable outside that context. It's a convenience so that we don't have to write an explicit if/then/else statement in the task output declaration. I don't think it's a good idea to apply elsewhere.length(select_all(...))
seems like a bug I need to look into further to #614 #615Taking a look at the SPEC again, it does look like coercion to None for nonexistent optional files is scoped to task outputs. So my previous first complaint isn't really an issue. However, would it be a good idea to apply this behavior for workflow outputs as well?
I think the length(select_all(...))
issue might boil down to needing more clarification from the WDL spec. For one, the working directory for string to file coercion sounds like it should be different depending whether in a task input, body, or output:
https://github.com/openwdl/wdl/blob/caff59db192636d9f93f3f5659eb5939f51ff877/SPEC.md?plain=1#L3944
If converting File?
types to None
happens at output, then running select_all
on an array with nonexistent coerced files will always return something different if not ran at the output step. For now, this behavior seems to be what the spec wants, but sounds incorrect as those File?
types should be None regardless. I'll probably open an issue on openwdl for this.
When a file is declared as optional, the value should be null if the file does not exist. In an array of optional files, this should mean those nonexistent file entries should become null.
When coercing from string to file, this check for file existence seems to be inconsistent depending on how the WDL is written:
With
MINIWDL__FILE_IO__ALLOW_ANY_INPUT=True miniwdl run test.wdl
, this returns:All of
len_in_body_out
,len_in_input_out
, andlen_in_output
should be 0, but when processed inside the task body/input, select_all runs on the string representation instead of the file representation; only processing it in the output does it check that those file paths exist.String to file coercion also behaves differently depending if it is within a task or a workflow; the above WDL is mostly in a task, as the same code does not work in a workflow: