search

BCV-Uniandes / AUNets

Pytorch implementation of Multi-View Dynamic Facial Action Unit Detection, Image and Vision Computing (2018)
MIT License
149 stars 30 forks source link

GPU memory error #29

Open malkaddour opened 4 years ago

malkaddour commented 4 years ago

Hello, I've been able to set up the entire BP4D dataset for training using the correct txt files. However, a memory error always comes up after preprocessing. My GPU memory is capped at 8.1 GB, and even when I try decreasing the batch size down to 4, it yields the same error. The code output is (batch size set to 4) "RuntimeError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 7.93 GiB total capacity; 7.22 GiB already allocated; 14.06 MiB free; 7.42 GiB reserved in total by PyTorch)".

I receive the same error even when I set the batch size down to 1, and monitoring my GPU RAM I see that it reaches close to it's capacity before the error. I also tried running it without GPU on a 126 GB RAM server, and it steadily increased RAM usage until the training was "killed".

Do you know if there is something in the training scheme that I should change to prevent this? Many thanks in advance, and for taking the time to read this.

MStumpp commented 4 years ago

Did you figure this out? Same problem.