search

ryanxingql / stdf-pytorch

Implementation of "Spatio-Temporal Deformable Convolution for Compressed Video Quality Enhancement" (AAAI'20).
https://www.aiide.org/ojs/index.php/AAAI/article/view/6697
Apache License 2.0
155 stars 20 forks source link

CUDA out of memory #18

Closed FloBEAUG closed 2 years ago

FloBEAUG commented 2 years ago

Hello

I'm trying to train MFQEv2_R3_enlarge300x but it fails near iter 5000 with the following error :

RuntimeError: CUDA out of memory. Tried to allocate 1000.00 MiB (GPU 0; 7.79 GiB total capacity; 5.21 GiB already allocated; 768.88 MiB free; 5.95 GiB reserved in total by PyTorch)

I have a MSI RTX 2080 VENTUS V2 with 8Go of RAM

Is there an option to limit the amount of used memory ?

ryanxingql commented 2 years ago

Hi I guess the program is trying to allocate more gpu memory for testing. You may turn off the regular testing, or conduct the testing on another gpu.

FloBEAUG commented 2 years ago

I have commented # CUDA_VISIBLE_DEVICES=0 python test.py --opt_path option_R3_mfqev2_1G.yml on the script.sh file but there is no change of behavior.

I just have one GPU on my computer which is used by the OS (linux mint 20.3). With all application closed the system is using an incompressible 500M of GPU memory which I think is causing my problem.

I will try to run in full console mode to free more memory but just in case, can I change some values in the option_R3_mfqev2_1G.yml file to reduce memory consumption ?

FloBEAUG commented 2 years ago

Nope, tried in full console environment with 0MB of memory used when launching the training script. Same problem.

ryanxingql commented 2 years ago

hi, sorry for late response

I mean you may turn off the regular validation in the training process. Check this line: https://github.com/ryanxingql/stdf-pytorch/blob/9021fd79962f9820295cf289ecc177dd12275530/train.py#L403

FloBEAUG commented 2 years ago

Tank you, it seems to work if i do that. I think my GPU has no sufficient memory to load 1080p video files.

I tried test the pre-trained models but it fails at the same point. Also tried test_one_video.py which works fine with the default video (BasketballPass_416x240_500) but when I try to open a 1080p video I got the same error.

ryanxingql commented 2 years ago

Glad to hear that!

If you cannot enhance videos with big resolution, you may cut each frame into smaller patches, enhance these patches and finally combine them into one frame.

Or, you may conduct the test process on cpu, which may be 10 times slower than that on gpu.

FloBEAUG commented 2 years ago

Ok thanks I will try that.