mlflow stopped responding

[RuiFilipeCampos]

I'm not entirely sure what happened

gonna deploy a more robust solution:

• docker compose
  • traefik as reverse proxy
  • mlflow container (scalable processes)
  • external database
• deployment
  • within the same subnet as the milan gpu machines
  • automated with github actions
  • cloud watch setup

on the side of the the gpu machine, I might code a worker to offload the metrics logging to a separate process, but hopefully having them in the same subnet will decrease latency to acceptable levels

[RuiFilipeCampos]

https://www.restack.io/docs/mlflow-knowledge-mlflow-dataset-management