search

junyanz / VON

[NeurIPS 2018] Visual Object Networks: Image Generation with Disentangled 3D Representation.
http://von.csail.mit.edu
Other
530 stars 90 forks source link

CUDA out of memory issue on training 2D Texture Network #27

Closed scamper07 closed 4 years ago

scamper07 commented 4 years ago

Hi, Thank you for the code and the detailed installation procedure.

I needed some information on the training steps for the car use case. We have the Quadro RTX 4000 GPU with 8 GB GDDR6 GPU memory. But on running the training for 2D texture network, it quickly uses up the memory and i have CUDA out of memory issue. I kept reducing the dataset and finally was able to give only 10 images for training.

What was the hardware that you used for training and did you do any extra config for training 2605 images for the texture network? Also can you suggest any steps to increase dataset without hitting the GPU memory issue. I'm using PyTorch 0.4.1 with CUDA92 within conda

Thank you

ztzhang commented 4 years ago

Hi, what batch size are you using and what are the resolution of your training images? The size of the dataset is not relevant to memory usage since only batches of them are loaded each time.

scamper07 commented 4 years ago

I'm using the default batch size i.e 12. I'm trying to train with the same car dataset so the resolution is the same.

ztzhang commented 4 years ago

I think most of our training is done with a 12GB RAM GPU. It should be fine if you lower the batch size, to 10 for example.

junyanz commented 4 years ago

Yeah, I will try 8 to 10.

scamper07 commented 4 years ago

Thank you so much. Batch size of 8 worked