sematic-ai / sematic

An open-source ML pipeline development platform
Other
975 stars 59 forks source link

Allow disabling k8s job cleanup #1115

Open augray opened 8 months ago

augray commented 8 months ago

When a run terminates (successful or unsuccessful), Sematic cleans up the corresponding k8s job. Sometimes when diagnosing problems with k8s, it can be helpful to look at job/pod states even after termination. We should allow a feature flag whereby you can disable job cleanup for debugging.