search

NVlabs / instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more
https://nvlabs.github.io/instant-ngp
Other
16k stars 1.93k forks source link

failed with error CUDA_ERROR_OUT_OF_MEMORY #1108

Open TK0627 opened 1 year ago

TK0627 commented 1 year ago

14:16:08 SUCCESS Loaded 20 images after 0s 14:16:08 INFO cam_aabb=[min=[0.617572,-0.877056,0.500298], max=[1.81439,1.80589,0.529203]] 14:16:08 INFO Loading network config from: configs\nerf\base.json 14:16:08 INFO GridEncoding: Nmin=16 b=1.51572 F=2 T=2^19 L=16 Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. Warning: FullyFusedMLP is not supported for the selected architecture 61. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+. 14:16:08 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1 14:16:08 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3 14:16:08 INFO total_encoding_params=13074912 total_network_params=10240 Attempting to free memory arena while it is still in use. Could not free memory arena: C:\ngp\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:489 cuMemAddressFree(m_base_address, m_max_size) failed with error CUDA_ERROR_INVALID_VALUE 14:16:08 ERROR Uncaught exception: C:\ngp\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:592 cuMemSetAccess(m_base_address + m_size, n_bytes_to_allocate, &access_desc, 1) failed with error CUDA_ERROR_OUT_OF_MEMORY

kindly help on this one

GPU:

Screenshot 2022-12-02 at 10 13 04
superhandishere commented 1 year ago

hardware configuration too low

TK0627 commented 1 year ago

@superhandishere what would be the minimum configuration or recommended configuration.

superhandishere commented 1 year ago

20 images dataset which is a reasonable figure, how about the resolution of each image?

TK0627 commented 1 year ago

20images with a resolution of 1080x720 also tried 300x300 with aab scale 4 and 1 respectively but same error

superhandishere commented 1 year ago

have you tried originial fox folder of images? does that work?

TK0627 commented 1 year ago

No, not yet tried the fox images.

superhandishere commented 1 year ago

try that first, if you can't run successfully then you have to upgrade hardware, recommend rtx30x

TK0627 commented 1 year ago

okay i will try to render with fox image and see, Thankyou.

fkay1 commented 1 year ago

I Have a 1070 with 8GB of VRAM and had the same issue with my own dataset. It turns out i had to decrease the "--aabb_scale" now the custom dataset works too.

AKanoniOS commented 1 year ago

@FGay

What did you lower it to?

superhandishere commented 1 year ago

Normally aabb scale to 4 is optimal (cuz fox data is set so)

doctorjana commented 1 year ago

okay i will try to render with fox image and see, Thankyou.

Did it work? I have the same config and I couldn't get it to work.

TK0627 commented 1 year ago

No it is not worked for me as well.

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

From: doctorjana @.> Sent: Saturday, December 17, 2022 9:22:42 PM To: NVlabs/instant-ngp @.> Cc: Thirukumaran Sankar @.>; Author @.> Subject: Re: [NVlabs/instant-ngp] failed with error CUDA_ERROR_OUT_OF_MEMORY (Issue #1108)

okay i will try to render with fox image and see, Thankyou.

Did it work? I have the same config and I couldn't get it to work.

— Reply to this email directly, view it on GitHubhttps://github.com/NVlabs/instant-ngp/issues/1108#issuecomment-1356316022, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4CVBALX5DYFOONV3XFJA53WNXOUVANCNFSM6AAAAAASRWGOAQ. You are receiving this because you authored the thread.Message ID: @.***>

doctorjana commented 1 year ago

No it is not worked for me as well. Sent from Outlook for Androidhttps://aka.ms/AAb9ysg … ____ From: doctorjana @.> Sent: Saturday, December 17, 2022 9:22:42 PM To: NVlabs/instant-ngp @.> Cc: Thirukumaran Sankar @.>; Author @.> Subject: Re: [NVlabs/instant-ngp] failed with error CUDA_ERROR_OUT_OF_MEMORY (Issue #1108) okay i will try to render with fox image and see, Thankyou. Did it work? I have the same config and I couldn't get it to work. — Reply to this email directly, view it on GitHub<#1108 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4CVBALX5DYFOONV3XFJA53WNXOUVANCNFSM6AAAAAASRWGOAQ. You are receiving this because you authored the thread.Message ID: @.***>

Looks like a common graphic card issue. Just hope someone finds a solution.