search

maxhodak / keras-molecules

Autoencoder network for learning a continuous representation of molecular structures.
MIT License
519 stars 146 forks source link

ResourceExhaustedError: OOM when allocating tensor with shape[600,120,1503] #49

Closed jeffrey9909 closed 7 years ago

jeffrey9909 commented 7 years ago

I am new to ubuntu and this program, so this question may be very simple... This error message shows when I run the train.py

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[600,120,1503]

I think that this may be my GPU running out of memory. My graphic card is a GTX970 with 4GB memory. I am running the program according to the README, which is inputting this line

python train.py data/processed.h5 model.h5 --epochs 20

I have read a very similar problem as below

http://stackoverflow.com/questions/39076388/tensorflow-deep-mnist-resource-exhausted-oom-when-allocating-tensor-with-shape

but I still cannot find a way to fix my problem. Can anyone please give me a hand? Thank you very much.

pechersky commented 7 years ago

In general, you'll probably get better performance on your machine is you use train_gen.py instead. You are getting an "out of memory" error because the default batch size is pretty big! (The repo developer runs a GTX1080). I have had my GTX970 work with a batch size of 300 iirc. So your command becomes python train_gen.py --batch_size 300 data/processed.h5 model.h5 --epochs 20.

jeffrey9909 commented 7 years ago

Oh, I see. Thanks for your help! I was thinking something like I need to change the code as the example I have mentioned. Thanks again! I will give it a try again.