openwdl / wdl

Workflow Description Language - Specification and Implementations
https://www.openwdl.org/
BSD 3-Clause "New" or "Revised" License
769 stars 307 forks source link

Require execution engines to support http(s) URLs in `File` declaration expressions #679

Open jdidion opened 3 months ago

jdidion commented 3 months ago

Currently, expressions in File declarations are required to evaluate to a path in the execution environment. We want to move away from allowing relative paths, so really the only use case for an expression in a file declaration is to specify an absolute path in the execution environment, which has limited utility.

It seems like a much more useful purpose for file declaration expressions would be to allow for localizing remotely available (public) resources. At a minimum we should require engines to support public http(s) URLs (i.e., not requiring authentication). We already require engines to support http(s) for imports so this doesn't seem like it would be a huge burden to put on engine developers.

patmagee commented 2 months ago

I think practically this makes a lot of sense, however I definitely know of cases where this would not be allowed (ie violataes service boundaries or environments cut off from external networks). Additionally, you may want to control where inputs are coming from to prevent external resources entering into a protected environment.

So far we have been fairly un-opinionated about the types of files an engine needs to support and have dealt largely in the abstract. Overall, I think this has served us well so I am hesitant to mandate support for a specific format