Improving the CWL DAG - Githubissues

unity-sds / unity-sps

The Unity SDS Processing Service facilitates large-scale data processing for scientific workflows.

Apache License 2.0

2 stars 2 forks source link

Improving the CWL DAG #107

Closed LucaCinquini closed 2 weeks ago

LucaCinquini commented 2 weeks ago

Purpose

This PR adds an improved version of the CWL DAG which a) does not use the Pod template (which would need to be copied to the DAGS folder separately) and b) uses the Karpenter node pool
Also added a few additional utilities: a Kubernetes deployment that can be used to mount the shared volume where the tasks write data, and a script to submit N DAGs

Proposed Changes

[ADD] New "cwl_dag_new" DAG that can execute a generic DAG on Karpenter node pools
[ADD] Utilities to monitor the shared volume and to submit N DAGs programmatically

Issues

https://github.com/unity-sds/unity-sps/issues/64

Testing

The CWL DAG was able to execute 10 concurrent DAGs of the "CWL echo" workflow.
Also able to execute the CWL DAG for the SBG PreProcess step