search

apax-hub / apax

A flexible and performant framework for training machine learning potentials.
MIT License
15 stars 2 forks source link

Input Pipeline rework #245

Closed M-R-Schaefer closed 6 months ago

M-R-Schaefer commented 6 months ago

I have reworked the data pipeline. The NL is no longer precomputed for the entire dataset and stored in memory. This used up a log of RAM and the conversion of unpadded NLs to tf ragged tensors was terribly slow. This rework drastically speeds up start up times and reduces memory consumption without compromising performance.