vocalpy / vak

A neural network framework for researchers studying acoustic communication
https://vak.readthedocs.io
BSD 3-Clause "New" or "Revised" License
78 stars 16 forks source link

ENH: Add extra "expt-tracking" that uses mlflow #778

Open NickleDave opened 1 month ago

NickleDave commented 1 month ago

I don't want to invest a lot right now in supporting every feature of mlflow but it's clear to me we need some sort of experiment tracking (as @achabotl wisely pointed out many moons ago).

Prefer going with mlflow for now since it's the one I'm most familiar with and I would rather just adopt something that's widely used in industry, although sth more lightweight and application-agnostic like sacred or sumatra might be worth thinking about

NickleDave commented 1 month ago

Just realized lightning has an mlflow logger https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.mlflow.html#mlflow-logger so we can perhaps somewhat painlessly integrate that way -- pass in experiment name, run name, etc

edit: looking at this again and related issues, questions in the Discussions, etc., my impression is that integration is not perfect and it's probably less painful in the long run to keep them separated if possible

edit edit: basically I need to implement the bad version of this and see if it can even work