search

riga / law

Build large-scale task workflows: luigi + job submission + remote targets + environment sandboxing using Docker/Singularity
http://law.readthedocs.io
BSD 3-Clause "New" or "Revised" License
98 stars 41 forks source link

Alternative high-level workflow implementation #137

Closed lmoureaux closed 1 year ago

lmoureaux commented 1 year ago

This is just a proof of concept.

See #131.

riga commented 1 year ago

Thanks for the draft! I'll have time early next week to dive into it.

choucavalier commented 1 year ago

Hi @riga

First of all, thanks for working on this awesome project.

I read this issue end ended up in this PR, and I am interested in this new API as well

Any progress on this?

Thanks

riga commented 1 year ago

I finally looked into this, and I think this approach would be the way to go. Unfortunately, as I wrote in the issue, the resolution from workflow parameters to branches, and vice versa would need to happen before the instance is actually created, so that parameters are correct directly from the start.

E.g. if certain workflow parameters would lead to the task being the workflow itself, it should be instantiated with branch=-1, and, vice-versa, if the workflow parameters would refer to a specific branch, this branch should be passed to the constructor to achieve a consistent setup.

I took parts of your approach, and added the parameter resolution to the class-level modify_param_values hook. I opened PR #159 for that, and will close this one.