search

globus-labs / mof-generation-at-scale

Create new MOFs by combining generative AI and simulation on HPC
MIT License
18 stars 5 forks source link

Load data onto device memory incrementally #85

Open WardLT opened 8 months ago

WardLT commented 8 months ago

The data loaders for Difflinker place the entire training set on device memory, which limits the size of datasets we can train on. We should instead change the train_step to move each batch to the device only when needed