While the recent input pipeline rework has already greatly reduced memory consumption, we can defuinitely decrease it further.
The current major RAM hog is the dict of ASE atoms, not the data pipeline. We can avoid this by reading the file on the fly in the generator instead of all at once, thereby avoiding the need to keep all atoms in memory.
While the recent input pipeline rework has already greatly reduced memory consumption, we can defuinitely decrease it further. The current major RAM hog is the dict of ASE atoms, not the data pipeline. We can avoid this by reading the file on the fly in the generator instead of all at once, thereby avoiding the need to keep all atoms in memory.