search

broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
972 stars 353 forks source link

Make Workflow Outputs copying work in a multi-backend / multi-auth world #1684

Open Horneth opened 7 years ago

Horneth commented 7 years ago

Currently Workflow outputs can be copied at the end of a workflow, and this is done by then engine WorkflowFinalizationActor. All the information this actor has is file paths as Strings. To be able to copy those files out it needs to create a Path from them with the right filesystem / auth, which it currently can't do reliably since it doesn't have any information about which backend produced this output or with which auth.

A possible fix to that would be to make wdl4s.WdlFile wrap java.nio.File instead of String, so the filesystem / auth information is not lost.

ruchim commented 7 years ago

@kcibul does this issue fall under the epic support for sub-workflows?

katevoss commented 6 years ago

@Horneth can you explain whether this impacts subworkflows or multi-cloud support, or both?

Horneth commented 6 years ago

The impact is on all workflows not just sub. The issue is that in a multi-backend world supporting different filesystems or authentication mechanisms it is likely that the current implementation of workflow outputs copying would break.