Adding autoscaling to the CWL DAG and SBG end-to-end step-by-step DAG - Githubissues

unity-sds / unity-sps

The Unity SDS Processing Service facilitates large-scale data processing for scientific workflows.

Apache License 2.0

2 stars 2 forks source link

Adding autoscaling to the CWL DAG and SBG end-to-end step-by-step DAG #114

Closed LucaCinquini closed 6 days ago

LucaCinquini commented 1 week ago

Purpose

This PR introduces autoscaling behaviour for 2 existing DAGs: the "cwl_dag" and the "sbg_L1_to_L2_e2e_cwl_step_by_step_dag"

Proposed Changes

[CHANGE] Modify "cwl_dag" to use Karpenter for autoscaling
[CHANGE] Modify "sbg_L1_to_L2_e2e_cwl_step_by_step_dag" to use Karpenter for autoscaling

Issues

https://github.com/unity-sds/unity-sps/issues/64

Testing

After the changes, both the "cwl_dag" and the "sbg_L1_to_L2_e2e_cwl_step_by_step_dag" completed successfully (single execution only).