search

unity-sds / unity-sps

The Unity SDS Processing Service facilitates large-scale data processing for scientific workflows.
Apache License 2.0
2 stars 2 forks source link

Improve support for DAG updates #130

Open LucaCinquini opened 5 days ago

LucaCinquini commented 5 days ago

Currently, the OGC API allows to register a DAG that already exists in a Github repo (which is periodically synced to the SPS DAG catalog). But if the DAG is updated in the Github repo, the updates are not automatically propagated to SPS: the DAG needs to be unregistered and then registered again to be updated.

We need to design and implement a better process for automatically propagating DAG updates to the SPS Airflow deployment.

rtapella commented 5 days ago

Are you sure? Do you mean that you need to build some custom CD during development… ? Can Airflow have multiple versions of the same DAG deployed (e.g. for comparison testing)?

Also this is just for the DAG, not for the App Package/algorithm part, correct?

LucaCinquini commented 5 days ago

So right now Airflow does not support DAG versioning, you would need to include the version in the ID, such as: "my_dag_id_1", "my_dag_id_2" etc. And yes this ticket is just about DAGs and Airflow, not about Application Packages. If different versions of an application package are available at different URLs, they can both be executed through the same "cwl_dag" DAG.