mir-group / allegro

Allegro is an open-source code for building highly scalable and accurate equivariant deep learning interatomic potentials
https://www.nature.com/articles/s41467-023-36329-y
MIT License
327 stars 46 forks source link

Different numbers of # Epoch in output files #16

Closed ghost closed 1 year ago

ghost commented 1 year ago

Dear developers,

I found the epoch numbers in the output files are different in one training loop.

image

Is that normal and why does this happen ?

Thanks. Sean

Linux-cpp-lisp commented 1 year ago

Hi @seanwong16 ,

Those epoch numbers are strange... can you elaborate on what you mean by "different in one training loop", and on what your setup us here?

simonbatzner commented 1 year ago

This looks like a filesystem issue with things writing at different times? Can you repro?

simonbatzner commented 1 year ago

Closing.