search

microsoft / AI2BMD

AI-powered ab initio biomolecular dynamics simulation
MIT License
394 stars 46 forks source link

train process issue #11

Closed lingcon01 closed 2 weeks ago

lingcon01 commented 1 month ago

Dear author, Thanks for your sharing. I followed your settings to train VisNet on the Aspirin dataset, but it encountered NaN values at the 89th epoch, and the training was terminated. The energy MAE and force MAE on the test set were 0.225 and 0.453, respectively. Could you please advise on how to resolve this issue?

Image

ElwynWang commented 2 weeks ago

Such problems may be raised by various reasons. Strictly following the hyperparameters in the repo, the training process should work smoothly. I don't the specific environment and operations during your model training, but maybe you can try small lr, smaller models and longer warm up process.