search

chenfengxu714 / SqueezeSegV3

BSD 2-Clause "Simplified" License
223 stars 51 forks source link

GPU memory #5

Open Xiangxu-0103 opened 4 years ago

Xiangxu-0103 commented 4 years ago

Hi, when I run the code with the model SSGV321, it always prompts me that CUDA OUT OF MEMORY until I set the batch_size to 1. My GPU memory is 12G, TITAN XP. I want to know how much GPU memory you use.

chenfengxu714 commented 4 years ago

Hi, we train our model on eight Titan RTX (24G) GPUs. For SSGV321, the batch size is 2 for each GPU and it costs about 16G on each one. For SSGV353, the batch size is 1 for each GPU and it costs about 16G too. The reason that it needs many GPU memory is that the models consist of many tensor unfolding operation, the implementation of which in Pytorch is extremely memory-consuming during training. We are working at reducing the memory and speed now. Thanks for your interest.

lyhdet commented 3 years ago

Hi, I have the same problem. "CUDA OUT OF MEMORY", when batch_size > 1. and get "RuntimeError: cuda runtime error", when batch_size=1.

image

I am using the NVIDIA 2080ti with 11GB memory

lyhdet commented 3 years ago

I set the "cudnn.benchmark = False", solve the "CUDA runtime error".

Solacex commented 3 years ago

Hello, will there be a light version that requires less GPU memory? really looking forward to it.

Stone-sy commented 3 years ago

RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1