NVlabs / instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more
https://nvlabs.github.io/instant-ngp
Other
16k stars 1.93k forks source link

failed with error CUDA_ERROR_NOT_ SUPPORTED #193

Closed Vovosunt closed 2 years ago

Vovosunt commented 2 years ago

When trying to run the testbed with the provided data I get a runtime error as in the code below. I've tried all the example data as well as the no_gui tag but alas to no success. I'm using an NVIDIA 1070 on my home machine so I suppose it might just not be up to the task? (From what I gathered from other threads, memory should not be an issue with the newest builds even with the CutlassMLP fallback)

10:18:18 SUCCESS  Loaded 50 images of size 1080x1920 after 0s
10:18:18 INFO       cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00
721,1.41295]]
10:18:19 INFO     Loading network config from: configs\nerf\base.json
10:18:19 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 61. Fallin
g back to CutlassMLP. For maximum performance, raise the target GPU architecture
 to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Fallin
g back to CutlassMLP. For maximum performance, raise the target GPU architecture
 to 75+.
10:18:19 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,l
ayers=3)]-->1
10:18:19 INFO     Color model:   3--[SphericalHarmonics]-->16+16--[FullyFusedMLP
(neurons=64,layers=4)]-->3
10:18:19 INFO       total_encoding_params=13074912 total_network_params=9728
10:18:19 ERROR    Uncaught exception: C:\Users\Vovo\Desktop\NVIDIA instant ngp\d
ependencies\tiny-cuda-nn\src\common.cu:63 cuMemGetAllocationGranularity(&granula
rity, &prop, CU_MEM_ALLOC_GRANULARITY_MINIMUM) failed with error CUDA_ERROR_NOT_
SUPPORTED
Tom94 commented 2 years ago

That's a strange error -- given that the codebase seems to work on GTX 1060 / 1080 (and even some 9XX cards), I would have expected virtual memory to not throw an error like this.

I've added a few extra checks in the latest version. Could you

git pull origin master
git submodule update --recursive

and try again / let me know which error you run into if any?

Could you also tell me the precise model of your GPU? (E.g. is it a max-Q GPU? Which vendor is it from?)

Vovosunt commented 2 years ago

Recloned, rebuilt fully through VS 2017 developer command prompt (was using cmake GUI previously) Still get a similar error. GPU is KFA2 GeForce GTX 1070 EX, 8 GB, regular consumer card, non-max-Q.

20:22:36 SUCCESS  Loaded 50 images of size 1080x1920 after 0s
20:22:36 INFO       cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00
721,1.41295]]
20:22:36 INFO     Loading network config from: configs\nerf\base.json
20:22:36 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 61. Fallin
g back to CutlassMLP. For maximum performance, raise the target GPU architecture
 to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 61. Fallin
g back to CutlassMLP. For maximum performance, raise the target GPU architecture
 to 75+.
20:22:36 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,l
ayers=3)]-->1
20:22:36 INFO     Color model:   3--[SphericalHarmonics]-->16+16--[FullyFusedMLP
(neurons=64,layers=4)]-->3
20:22:36 INFO       total_encoding_params=13074912 total_network_params=9728
20:22:36 ERROR    Uncaught exception: GPU 0 does not support virtual memory.
Tom94 commented 2 years ago

@Vovosunt the latest commit https://github.com/NVlabs/instant-ngp/commit/d5fc14120f625923609647606e2ed47df095f631 contains a fallback option when virtual memory doesn't work. It'll use a bit more memory and may stutter on startup, but hopefully will otherwise be smooth.

It's important to also update submodules, or the fix won't do anything:

git submodule update --recursive

Could you check and report back? Thanks!