search

atomistic-machine-learning / schnetpack

SchNetPack - Deep Neural Networks for Atomistic Systems
Other
774 stars 214 forks source link

Fluctuations in Training Loss Functions #639

Closed MasterLucas closed 3 months ago

MasterLucas commented 3 months ago

Dear Sir/ Madam,

I have been training a PaiNN model, and when analyzing the results, I noticed that the training loss oscillates significantly across iterations. I suspect this happens because the model does not compute the average loss across batches before displaying it. How can I solve this issue?

Yours faithfully,

Lucas Bandeira

jnsLs commented 3 months ago

Dear Lucas, are you using the pytorch-lightning trainer? https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.trainer.trainer.Trainer.html#lightning.pytorch.trainer.trainer.Trainer

The model weights are updated after each iteration. If training seems to be unstable you could reduce the learning rate, increase the batch size or you could set accumulate_grad_batches to a desired number of batches.

Best, Jonas

MasterLucas commented 3 months ago

Dear Jonas,

Thank you very much for your answer. I changed the batch size and it seems to have solved the problem. Again, thank you very much.

Yours sincerely,

Lucas Bandeira