Closed VictorLlu closed 4 years ago
I also encountered the same problem with the default Cityscapes config on my 4 * V100 (16 GB mem each) instance.
We ran our experiments on 8x 32GB V100 GPUs. In order to be able to train with 16GB memory you will need to lower the working resolution, e.g. by reducing the value of longest_max_size
. Setting longest_max_size=2048
and random_scale = (0.5, 1.0)
should probably work with your setup, but it will not reproduce the results from our paper.
I use 8 gpus. Each one has 16GiB capacity, but the program still meets this problem. RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 5; 10.92 GiB total capacity; 6.01 GiB already allocated; 1.74 GiB free; 2.62 GiB cached). How can I fix this problem