Develop luca - Githubissues

unity-sds / unity-sps

The Unity SDS Processing Service facilitates large-scale data processing for scientific workflows.

Apache License 2.0

2 stars 2 forks source link

Develop luca #95

Closed LucaCinquini closed 3 months ago

LucaCinquini commented 3 months ago

Purpose

The purpose of this PR is to improve the DAG that is used to test scalability with Karpenter. Specifically:

The DAG id is the same as the Python file name, except for the extension
The Docker image used is "busybox" which allows for configurable runtimes
The affinity dictionary is obtained through a standalone function

In addition, the GitSync process to sync with the remote DAG repository points to the "develop" branch by default.

Proposed Changes

[CHANGE] Update the karpenter_test.py as outlined above
[FIX] Use "develop" as the default branch to git-sync the DAGs

Issues

https://github.com/unity-sds/unity-sps/issues/64

Testing

Executed 10 concurrent "karpenter_test" DAG runs, they all succeeded. Confirmed that 10 new nodes are created by Karpenter for each task, then deleted.