sematic-ai / sematic

An open-source ML pipeline development platform
Other
969 stars 58 forks source link

Allow disabling k8s job cleanup #1115

Open augray opened 5 months ago

augray commented 5 months ago

When a run terminates (successful or unsuccessful), Sematic cleans up the corresponding k8s job. Sometimes when diagnosing problems with k8s, it can be helpful to look at job/pod states even after termination. We should allow a feature flag whereby you can disable job cleanup for debugging.