How to solve CUDA out of memory ?

[Michaelwhite34]

I am using gtx 1660ti with 6gb vram.

[Michaelwhite34]

Processing: 45576d11ffe4dd13222ffdd.jpeg Speed:20.1300, 1 / 2, 0.1049 0%| | 0/800 [00:01<?, ?it/s] Traceback (most recent call last): File "fit_images.py", line 414, in fit(args, device) File "fit_images.py", line 252, in fit rendered_img = faceverse_model.renderer.alb_renderer(mesh) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/renderer.py", line 62, in forward images = self.shader(fragments, meshes_world, kwargs) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, *kwargs) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/shader.py", line 262, in forward colors = flat_shading( File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/shading.py", line 219, in flat_shading ambient, diffuse, specular = _apply_lighting( File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/mesh/shading.py", line 33, in _apply_lighting light_specular = lights.specular( File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/lighting.py", line 280, in specular return specular( File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/pytorch3d/renderer/lighting.py", line 152, in specular reflect_direction = -direction + 2 (cos_angle[..., None] * normals) RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 5.80 GiB total capacity; 3.86 GiB already allocated; 52.88 MiB free; 3.90 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[Michaelwhite34]

Processing: 45576d11ffe4dd13222ffdd.jpeg Speed:5.5962, 1 / 2, 18.7387 0%| | 0/800 [00:00<?, ?it/s] Traceback (most recent call last): File "fit_images.py", line 414, in fit(args, device) File "fit_images.py", line 234, in fit detailimg, = g_detail([latent_detail], input_detail mask_detail, return_latents=True, noise=noises_detail) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "/home/michael/Downloads/FaceVerse-main/network.py", line 648, in forward out = conv1(out, latent[:, i], noise=noise1) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "/home/michael/Downloads/FaceVerse-main/network.py", line 360, in forward out = self.conv(input, style) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "/home/michael/Downloads/FaceVerse-main/network.py", line 258, in forward weight = self.scale self.weight style RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.80 GiB total capacity; 2.06 GiB already allocated; 57.94 MiB free; 2.11 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[Michaelwhite34]

Someone please help -- Really struggling to get this working

[LizhenWangT]

Maybe you can try to comment out all the code related to the expression network. (Line 291 # fitting with the exp ckpt & the model loading started from Line 76)

[Michaelwhite34]

Maybe you can try to comment out all the code related to the expression network. (Line 291 # fitting with the exp ckpt & the model loading started from Line 76)

Should I just delete fit_images line 291 to line 341 or just comment out these lines with exp ? After deleting fit_images line 291 to line 341 and comment out line 73 to line 76

Processing: 45576d11ffe4dd13222ffdd.jpeg Speed:5.4946, 1 / 2, 18.7387 lm: 15.3111; pho: 0.6716; noise: 0.0017; : 1%| | 7/800 [00:05<09:44, 1.36it/s Traceback (most recent call last): File "fit_images.py", line 365, in fit(args, device) File "fit_images.py", line 265, in fit loss.backward() File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/autograd/init.py", line 154, in backward Variable._execution_engine.run_backward( File "/home/michael/anaconda3/envs/quick_test_env/lib/python3.8/site-packages/torch/autograd/function.py", line 199, in apply return user_fn(self, *args) File "/home/michael/Downloads/FaceVerse-main/third_libs/stylegan_ops/fused_act.py", line 80, in backward grad_input, grad_bias = FusedLeakyReLUFunctionBackward.apply( File "/home/michael/Downloads/FaceVerse-main/third_libs/stylegan_ops/fused_act.py", line 32, in forward grad_input = fused.fused_bias_act( RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 5.80 GiB total capacity; 3.54 GiB already allocated; 110.44 MiB free; 3.97 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF