search

insitro / redun

Yet another redundant workflow engine
https://insitro.github.io/redun/
Apache License 2.0
517 stars 43 forks source link

[EXPLORE] Exploring moving File staging logic to Executors #69

Closed mattrasmus closed 4 months ago

mattrasmus commented 1 year ago

Hi @msamman, here is an exploration branch trying out moving the File staging logic to Executors, by delaying the rendering of the shell code until just before submission. This is another branch based on #66. This allows us to have the Executor config (in addition to task options) decide whether to use the as_mount=True mode for File staging.

One advantage of this is allow us to not need to do script(..., executor="batch", as_mount=True) and instead define as_mount = True in the redun.ini for the executor. One reason this is beneficial is in cases where one wants to port a workflow from between batch systems. Ideally, they should only need to change the redun.ini file and not the workflow code itself.

This is a deeper library change, but I'm starting to see the benefits of delaying staging rendering so that Executors can customize it. I imagine this would be useful to do for future Executors as well.

danielecook commented 1 year ago

It would be nice to have the flexibility to stage by copying in or stage by mount - but I imagine you could configure two different 'executors' (actually running on the same infrastructure) for that use case.

mattrasmus commented 1 year ago

It would be nice to have the flexibility to stage by copying in or stage by mount - but I imagine you could configure two different 'executors' (actually running on the same infrastructure) for that use case.

Yep. The Executors defined in the redun.ini can reference the same infrastructure but with just different config (as_mount, or image, etc). You can also still customize at the call-site using script(..., as_mount=True).

mattrasmus commented 4 months ago

Many of these ideas have been incorporated since.