Memory

[spot92]

I have a 1080 ti with 11 GB of VRAM and I am running into this:

RuntimeError: CUDA out of memory. Tried to allocate 148.00 MiB (GPU 0; 11.00 GiB total capacity; 8.73 GiB already allocated; 44.32 MiB free; 77.81 MiB cached)

I am doing literally nothing else with my computer except styling an image, but it seems that the size I can perform is rather small given my "available" VRAM. I put available in quotes because I know it exists, but I am uncertain about why there is such a low amount that is actually free.

[ProGamerGov]

Are you using -backend cudnn? Using a VGG-16 or NIN model also helps lower memory usage, in addition to using -optimizer adam.

[spot92]

Yeah, I messed around with the options. Trying to figure out what is using up so much of my memory as I get this error:

RuntimeError: CUDA out of memory. Tried to allocate 148.00 MiB (GPU 0; 11.00 GiB total capacity; 8.73 GiB already allocated; 44.32 MiB free; 77.81 MiB cached)

Not sure what is using 8.73 gigs as the style transfer is the only thing I'm doing. Thoughts?

[ProGamerGov]

I can normally get above an image size of 1664px with the default model and L-BFGS on a 11-12GB graphics card, so I'm not quite sure what's causing your error. https://github.com/ProGamerGov/neural-style-pt/issues/3#issuecomment-554084645 There aren't any memory leaks in the code that I know of, and I've never experienced a memory error like what your describing before.

Can you please give me some more details about the parameters that you are using? Like specifically -image_size, -optimizer and the other really resource intensive parameters?

Also, have you tried reinstalling PyTorch and neural-style-pt?

[spot92]

I was able to successfully do a 1664X936 image by using the cudnn, cudnn_autotune, and adam. Will push it to see what the max is. But when I ran it with these parameters, I got the following message:

UserWarning: Matrix multiplication for dimensions larger than 2^21 has known bugs on your combination of CUDA version and device type. Please consider upgrading to CUDA 9.2 or later.

I just checked my version of CUDA and I am at 9.2. Going to check the PyTorch version now.

[spot92]

I have torch 1.3.1

[spot92]

My max is somewhere around an output size of 1900X1068 for those curious.

[ProGamerGov]

What's your OS? And driver version?

[ProGamerGov]

So, I just tested the following on a Tesla K80 with 12 GB (11441MiB) of memory, and it worked without running out of memory:

python3 neural_style.py -init image -tv_weight 0 -backend cudnn -image_size 1664 -optimizer lbfgs -model_file models/vgg19-d01eb7cb.pth

That's on the lastest version of PyTorch, and CUDA 10. Is there any reason that you are still on CUDA 9.2?

[spot92]

I am using CUDA 9.2 mostly because I have been way too lazy to update, but I'll do that. I was able to get to the 1664 image, I think I just put too much faith in my 1080 ti to go bigger.