search

apax-hub / apax

A flexible and performant framework for training machine learning potentials.
MIT License
15 stars 2 forks source link

fixed memory leak in pbp ds #290

Closed M-R-Schaefer closed 3 months ago

M-R-Schaefer commented 3 months ago

by reinitializing the processpoolexcecutor every couple of epochs, we avoid a memory leak. Not entirely sure what causes it in the first place but this is the simplest solution I can think of. While reinitializing every epoch is perfectly fine for large datasets, small datasets can get a slight performance hit when restarting the processes all the time. Since even MP/SPICE can go for more than 10 epochs just fine, I think it's okay to hardcode it to 10.

M-R-Schaefer commented 3 months ago

Should I test this on SPICE training to be sure it works?

sounds like a good idea!