Reset val/test metrics and losses after every epoch

Genentech / gReLU

gReLU is a python library to train, interpret, and apply deep learning models to DNA sequences.

https://genentech.github.io/gReLU/

MIT License

230 stars 23 forks source link

Reset val/test metrics and losses after every epoch #8

Closed suragnair closed 4 months ago

suragnair commented 5 months ago

Reset val_losses, val_metrics, test_losses, test_metrics at the end of val/test set so that it does not carry forward to subsequent epochs, as this will not reflect the loss in the latest epoch.

gokceneraslan commented 4 months ago

Cool fix! Any comments on the failed tests?

suragnair commented 4 months ago

seems to happens on CPU and not on GPU
model training on GPU seems to be normal
error happens due to trainer.validate(model=self, dataloaders=val_dataloader) being called before starting training
It also has to do with metrics being edited in "InferenceMode" in the trainer.validate step, and then it is not possible to update them without torch.inference_mode() context