coqui-ai / Trainer

🐸 - A general purpose model trainer, as flexible as it gets
185 stars 109 forks source link

Update trainer.py #125

Closed meryemsakin closed 9 months ago

meryemsakin commented 1 year ago

Fix for TypeError During Training

This fix addresses an issue where a TypeError occurred during training.

The problem was that during training, when tracking the best loss (self.best_loss), there were cases where the loss value unexpectedly came in the form of a dictionary. This dictionary contained multiple values, such as "train_loss" and "eval_loss," causing subsequent comparisons to raise a TypeError.

This fix consists of two steps:

Updating the self.best_loss variable with ch["model_loss"] to ensure it contains the training loss value. Correcting the self.best_loss variable to only contain the "train_loss" value. The aim of this fix is to accurately track the best training loss value during the training process.

erogol commented 11 months ago

Thanks for the PR. But we should follow the best eval loss if the model is running evaluation.

akx commented 10 months ago

You should probably figure out why you're unexpectedly getting a dictionary instead...