search

mapillary / seamseg

Seamless Scene Segmentation
BSD 3-Clause "New" or "Revised" License
289 stars 53 forks source link

CUDA out of memory #19

Closed VictorLlu closed 4 years ago

VictorLlu commented 4 years ago

I use 8 gpus. Each one has 16GiB capacity, but the program still meets this problem. RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 5; 10.92 GiB total capacity; 6.01 GiB already allocated; 1.74 GiB free; 2.62 GiB cached). How can I fix this problem

txfs1926 commented 4 years ago

I also encountered the same problem with the default Cityscapes config on my 4 * V100 (16 GB mem each) instance.

ducksoup commented 4 years ago

We ran our experiments on 8x 32GB V100 GPUs. In order to be able to train with 16GB memory you will need to lower the working resolution, e.g. by reducing the value of longest_max_size. Setting longest_max_size=2048 and random_scale = (0.5, 1.0) should probably work with your setup, but it will not reproduce the results from our paper.