train and validation metrics

[erwallace]

Hello, where/how can I specify multiple metrics I want to track during training for the train and validation datasets? And if I create custom metrics and specify them as plugins can I use any of those as well. Cheers

[wardhaddadin1]

Hello!

Currently, the only things tracked are losses and gradients.

It would be good to have an explicit example to understand a bit more the utility vs complexity of making something like metric computation configurable. Do you have an example?

Thanks, Ward

[erwallace]

Hi Ward, I'd like to be able to track the MAE and RMSD of the energies and forces independently for each epoch using the validation set when training a MACE model. Adding metrics for the train dataset was just out of curiosity - losses and gradients will suffice there. Thanks, Ewan

[wardhaddadin1]

Ah I see! That's doable. We can easily make the lighting log each of the losses and the total loss separately (I meant to do that, but forgot actually).

Tracking multiple metrics (not losses) per quantity is a bit trickier (will need some thought), but is a fair ask.

Would it unblock you for now if we track the losses for each quantity alongside the total loss for now? (and try to think more about how to do general metric tracking for a future PR?)

[erwallace]

Hi Ward,

Separate losses is a good start and would be appreciated. I’m also very happy to work with you on a PR for other metrics to speed up the process.

Thanks

[wardhaddadin1]

Cool! PR is up!

And thank you that's great! I think one way to do this would be to follow a similar direction to how losses are implemented in physicsml. They are generalised wrappers around standard torch losses which can compute the losses directly on each model output (y_graph_scalars, y_node_vector, etc...). Something like this, but with an additional option to specify multiple losses per output would work I think?

Best, Ward