Investigate and use Dag Factory for Roger workflow

oihawkins commented 2 years ago

Implement ways to generate workflows dynamically through a restful endpoint.
User submits minimal workflow declarations as a yaml. An Airflow dag is constructed on the fly and run. One approach to this is using Dag factory. Based discussions with @stevencox , it sounds like a reasonable approach might be to template this with some templating engine (Jinja) and have a minimalistic data structure for Workflow definition.

oihawkins commented 2 years ago

View historical comments for this ticket https://github.com/helxplatform/development/issues/584

YaphetKG commented 2 years ago

Just as an idea around this, we are exploring options around using a more robust data management repository as part of the new architecture for dug. There are two options currently at the table we are investigating:

IRODS
Lakefs When making a more data driven deployments, we need to be able to customize the DAGs in airflow with zero clicks. I.e Push in things like dataset commit ids and other provenance related items , to make sure that DAG runs also retain the provenance making tracking of what dataset run against which code.

YaphetKG commented 2 years ago

Moved to Jira (https://renci.atlassian.net/jira/software/projects/DUG/boards/2/backlog?selectedIssue=DUG-71)

helxplatform / dug

Investigate and use Dag Factory for Roger workflow #219