Training on big image get cuda runtime error,out of memory(nvidia 1070 8g)

leeyeehoo / CSRNet-pytorch

CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes

647 stars 261 forks source link

Training on big image get cuda runtime error,out of memory(nvidia 1070 8g) #47

Open Devin19910617 opened 5 years ago

Devin19910617 commented 5 years ago

I use this model to auto count object total number,when image is 4000x4000,train will get error 'out of memory',I don't want to downscale image size,What should I do?

kurtabela commented 3 years ago

Same thing happens when I try to insert an image of 4096*2304, of size 1.3MB. For some reason it tries to allocate 2.25GiB...

RuntimeError: CUDA out of memory. Tried to allocate 2.25 GiB (GPU 0; 6.00 GiB total capacity; 2.54 GiB already allocated; 1.97 GiB free; 2.57 GiB reserved in total by PyTorch)