search

1adrianb / face-alignment

:fire: 2D and 3D Face alignment library build using pytorch
https://www.adrianbulat.com
BSD 3-Clause "New" or "Revised" License
6.89k stars 1.33k forks source link

CUDA out of memory #250

Closed moncio closed 3 years ago

moncio commented 3 years ago

Hi, I'm discovering a problem sometimes at the first detection of the library. It seems the system in this first detection, it needs a large quantity of GPU memory coming the next error:

RuntimeError: CUDA out of memory. Tried to allocate 2.26 GiB (GPU 0; 10.76 GiB total capacity; 355.45 MiB already allocated; 2.27 GiB free; 2.38 GiB reserved in total by PyTorch

My computer is equipped by RTX 2080 ti with 11 GB (I'm executing other algorithms at the same time). My question is why the library is consuming so many GB at the beginning (Watching nvidia-smi command), I observe almost 8 GB but quickly decrease to 4 GB keeping this rate all the execution)

Thanks in advance

1adrianb commented 3 years ago

Hi @moncio The detector memory usage, in particular of SFD will depend on the resolution of input image. You can control/limit this by resizing your image prior to this. Note that some faces may be missed if they get to small. Alternatively, you can try the blaze face detector which should use less.

moncio commented 3 years ago

In this case, I'm using the BlazeFace and I tried to downsample and the issue continues...

1adrianb commented 3 years ago

If you already did that, in this case most likely its the cudnn benchmarking flag that during the first run will try different algorithms, see for example [1]. As such, you could warm-up the network prior to running the rest of your code. Disabling the benchmarking may lead to slower execution.