search

ashual / scene_generation

A PyTorch implementation of the paper: Specifying Object Attributes and Relations in Interactive Scene Generation
Apache License 2.0
185 stars 30 forks source link

Frequently CUDA out of memory #5

Closed zzilch closed 4 years ago

zzilch commented 4 years ago

Hi ashual, I tired to train the model from scratch. However, I always got a CUDA out of memory error after a few iterations (one at 14th epoch and another at 5th epoch). What's your GPU for training the model? I didn't find this information neither in the paper nor this repo. I used a GPU with 12G memory. I didn't come up with this problem when training the sg2im model with their offical model. I will try to use a smaller batch size. Did you store some gpu tensor somewhere won't be gc?

AmingWu commented 4 years ago

Hi ashual, Could you provide a code that supports multiple GPU parallelism?

ashual commented 4 years ago

@zzilch the problem is that each image contains 3-8 objects, that means that if your batch size is 10 you can get an object's batch of 30-80, which is a pretty big variance. Smaller batch size is indeed the simplest solution to this problem.

ashual commented 4 years ago

@AmingWu this is not a trivial task since the training begins with a batch size of the objects and continues with a batch size of the images.