search

openwdl / wdl

Workflow Description Language - Specification and Implementations
https://www.openwdl.org/
BSD 3-Clause "New" or "Revised" License
763 stars 307 forks source link

Semantics for `runtime` `disks` mount points are confusing #672

Open adamnovak opened 2 months ago

adamnovak commented 2 months ago

The disks key in the runtime section is documented as providing "persistent volumes" at certain mount point paths. Those paths are specified as being "in the host environment" in WDL 1.2, and I think "on the host machine" in 1.1. https://github.com/openwdl/wdl/blob/664adc323f536ab1fac0c41bf3fc85f4f15eb52b/SPEC.md?plain=1#L5072-L5088

If the mount point is a host side path, then where is the storage expected to be mounted in the container where the WDL command section gets run? If the storage is meant to be mounted at the given path in the container, then why does that path need to also exist on the host?

Is this meant to just let the WDL task get access to a particular directory on the host, by mounting that path on the host into the container at the same path?

What kind of "persistence" specifically is supposed to be available? If two tasks run one after the other, and they both have a disks entry with a given mount point and size, should the second task be guaranteed to see files there written by the first task? Or can the "persistent" volume be a fresh empty directory for each task? Or is some kind of opportunistic sharing expected?

If a task requests a 100 GiB persistent volume, does it have to deal with the possibility that, upon being mounted, it already had 50 GiB of files in it left over from previous tasks and only has 50 GiB free space?

If two tasks run at the same time, can they ever share a persistent volume? Or does a task get exclusive ownership of a persistent volume while it is mounted?

We're trying to implement this in Toil in https://github.com/DataBiosphere/toil/pull/5001 and so far we've come up with an implementation that just mounts the specified path from the host into the container. But I think it really makes more sense to mount fresh empty directories with the given amount of reserved space into the container instead, since that matches what I would imagine a workflow would actually want. But that completely ignores the "persistent" part of the spec.

Are there any workflow examples that use mount points, beyond the test examples in the spec that just measure their size? What kind of behavior do they expect w.r.t. persistence or the relationship between in-container and host-side paths?

adamnovak commented 2 months ago

Cromwell's implementation provides explicitly ephemeral storage ("All disks are set to auto-delete after the job completes.") and appears to mount the requested amount of storage at the requested path as seen from the task's perspective, with no reference to the host machines' filesystems.

jdidion commented 2 months ago

The specified mount points must be available to the task. So if the task is executing in a Docker container, the volumes must be mounted in the container. Whether and how those volumes map to volumes on the host machine is up to the execution engine.

I don't think we ever considered whether it's allowed for the mount point to already exist. I think the way Cromwell does it is probably the most logical. If the mount point already exists, it should be an error, although I guess it would be ok for the execution engine to decide that its ok to attach an existing, empty mount point that has at least the amount of requested storage.

We probably want to add a separate volume requirement that would cause an existing volume to be mounted read-only to allow accessing files on a shared file system, EBS volume, etc.