Add deployment variables to Airflow environment

unity-sds / unity-sps

The Unity SDS Processing Service facilitates large-scale data processing for scientific workflows.

Apache License 2.0

2 stars 2 forks source link

Add deployment variables to Airflow environment #91

Closed jpl-btlunsfo closed 1 month ago

jpl-btlunsfo commented 1 month ago

Purpose

Make relevant Unity environment variables available in the SPS Airflow deployment

Proposed Changes

ADD unity environment variables (project, venue, deployment name, counter, cluster name) to Airflow container environment
ADD simple environment printing DAG (only for testing, removable)

Issues

unity-sds/unity-sps#85

Testing

Tested with manual cluster
Used new env_test dag to expose available variables (not visible in Airflow UI)

LucaCinquini commented 1 month ago

I agree... I was in the process of testing but my cluster won't undeploy because the PV not being destroyed, something that needs fixing. But great job Brad :)

drewm-jpl commented 1 month ago

@LucaCinquini if that happens try force deleting the pods that have that PVC attached. It's a temporary way of dealing with that issue for now.

kubectl delete pod/airflow-worker-0 --grace-period=0 --force --namespace=airflow

LucaCinquini commented 1 month ago

The command worked well to destroy the Pod, but the SPS still wouldn't undeploy, so I am back to redeploying the EKS. Valiant effort though...

drewm-jpl commented 1 month ago

Do you have any other orphaned pods? You can try to inspect the PV/PVC? You can usually identify which pod is holding onto the PV and terminate it.

LucaCinquini commented 1 month ago

Ok, will check more next time. I went the nuclear option already.