search

NVIDIA / semantic-segmentation

Nvidia Semantic Segmentation monorepo
BSD 3-Clause "New" or "Revised" License
1.76k stars 388 forks source link

Impact of reducing crop_size to train on lighter gpus + original training times #119

Closed the-joy-of-data closed 3 years ago

the-joy-of-data commented 3 years ago

Hello,

Thank you for sharing your work.

I ran some eval tests as mentioned in the README and they went well, I get the exact same IoU of 86.92 for eval_cityscapes.yml

Next, I tried train_cityscapes.yml. I needed to bump down the crop size to 512, 1024 because I was running out of memory, and that worked as well. But now I also am thinking of re-training with lower sizes to reduce training time

I have two main questions -

  1. Please comment if this reduction in crop_size could have an impact on the model and accuracy.

  2. What were your training times when training the mapillary-pretrained model from scratch? What gpu architecture did you guys use?

ajtao commented 3 years ago
  1. Yes definitely, larger crop sizes help accuracy. However, an easy way to save some memory without sacrificing too much accuracy is to remove the RMI loss in favor of just cross entropy. That might allow you to use a larger crop size.
  2. Training the Mapillary model takes like 3-4 days on 4 nodes of 8gpus apiece (V100 i believe is what we used).
the-joy-of-data commented 3 years ago

Thank you for the response, yes I did go ahead and try with rmi loss removed.