[Feature]: RunAI overquota resilience

aisingapore / kapitan-hull

The one-stop shop of a Cookiecutter template to spin up a working AISG project in minutes.

https://aisingapore.github.io/kapitan-hull/

8 stars 0 forks source link

Open Syakyr opened 2 months ago

Syakyr commented 2 months ago

The codebase is more or less resilient against unexpected stops and restarts of job containers. But we can improve on it with:

Resumption on the same MLFlow experiment name after resuming
Rolling file handler for logs to reduce the reliance on looking at RunAI for the logs since it can start and stop at any time