Closed freehome1 closed 10 months ago
It seems that the memory keep rising when training like this:
and finally oom
Hi, we trained all the models without any problems on a GPU with much smaller RAM (GeForce 1070), so I am not sure why this error occurs in your case. You can try using a GPU with more RAM (if you have such a possibility) or reducing the batch size. However, the batch size we use is already small (4), and reducing it could lead to unstable training.
Best, Mikołaj
It's ok when I train on USPTO-50K and CUDA out of memory on USPTO-full![image](https://github.com/molecule-one/megan/assets/32356286/a4aadb8e-152c-4704-afa3-1ee2460973b0)