ulissigroup / amptorch

AMPtorch: Atomistic Machine Learning Package (AMP) - PyTorch
GNU General Public License v3.0
59 stars 35 forks source link

GMP feature with lmdb #88

Closed ray38 closed 2 years ago

ray38 commented 3 years ago

some major updates that we have:

    • no-caching (load directly from disk, very slow and inefficient, but support infinite amount of training data, not recommended)
    • full-caching (load lmdbs to memory first, very fast IO and training, tho amount of training data limited by CPU RAM, recommended if possible, ~100x faster than the no-caching way)
    • partial-caching (partially load lmdb into memory during training, close to full-caching speed but support infinite amount of training data. ~2-3x slower than full-caching, but ~30-50x faster than no-caching)