RuntimeError: CUDA out of memory while training

JDAI-CV / Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving

Virtural try-on under arbitrary poses

MIT License

217 stars 65 forks source link

RuntimeError: CUDA out of memory while training #21

Closed iamrishab closed 4 years ago

iamrishab commented 4 years ago

Hi! Can you please tell me what hardware configuration are you using for training? Especially for Step 4 and Step 5:

### Step 4: Train Face Refinement
python train.py --train_mode face
### Step 5: Train jointly
# python train.py --train_mode appearance --joint_all --joint_parse_loss

Currently, I am using 8 K80s but still getting CUDA out of memory while training. Thanks!

AIprogrammer commented 4 years ago

We use 4 x Tesla P40 while training, maybe you can reduce the batch size.

iamrishab commented 4 years ago

Yes, I was changing a different batch (batch_size_v) which is not used in the dataloader. Changing batch_size_t actually worked. Thanks!

iamrishab commented 4 years ago

We use 4 x Tesla P40 while training, maybe you can reduce the batch size.

Can you please share the details system configuration for model training using default hyperparemeters in the config.py? tia

iamrishab commented 4 years ago

Never-mind. I figured it out. Thanks!