openwdl / wdl

Workflow Description Language - Specification and Implementations
https://www.openwdl.org/
BSD 3-Clause "New" or "Revised" License
776 stars 307 forks source link

Undefined behavior for string to file coercion when file does not exist #667

Open stxue1 opened 4 months ago

stxue1 commented 4 months ago

It's unclear if a string being coerced to a file should error or not depending on if the file exists and that the file is not outputted in the WDL.

For example:

version 1.1
workflow test_map {
  File f = "path/to/file"

  output {
  }
}

It's unclear whether this workflow should error on File f if the file does not exist.

MiniWDL seems to not error on these cases, but only errors if the file is to be outputted, for example:

version 1.1

workflow test_map {
  File f = "path/to/file"

  output {
    File out = f
  }
}

A lot of the SPEC tests also depend on this behavior, for example: https://github.com/openwdl/wdl/blob/664adc323f536ab1fac0c41bf3fc85f4f15eb52b/SPEC.md?plain=1#L1190-L1208

In particular, this section about type coercion talks about string to file coercion, but doesn't mention whether the file must exist at the coercion step.

Should execution engines ensure that files must exist at coercion or only at a workflow/task output?

patmagee commented 4 months ago

As of now there is no requirement in the spec for a file declaration to actually exist. You can declare any string as a file, it will simply fail when the engine tries to localize it or read it.

This behaviour is consistent with other programming languages which allow files to be declared without enforcing their existence

jdidion commented 4 months ago

Agree w @patmagee. The question is whether we want to make this explicit in the spec. I think we should.