Could not initialize Vulkan and NGX. DLSS not supported

jaggzh commented 2 years ago

Linux, Debian stable Git build from this morning.

I actually had instant-ngp working on this system months ago, but now get only errors. My system has a 1050 Ti (4gb) and two K80's (one K80 has two 12gb cards). (There could have been other changes, like nvidia drivers maybe put an update in debian or something), but I had it working at commit d793984b0). I've seen some similar errors in other posts, but I'm not sure if it's the same issue.

With the two types of cards, the K80's come after the 4gb 1050, so I use: export CUDA_VISIBLE_DEVICES=1,2

19:36:32 INFO Loading NeRF dataset from
19:36:32 INFO data/pulseox/transforms.json
19:36:32 SUCCESS Loaded 98 images after 0s
19:36:32 INFO cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:36:33 INFO Loading network config from: configs/nerf/small.json
19:36:33 INFO Loading parent network config from: configs/nerf/base.json
19:36:33 INFO GridEncoding: Nmin=16 b=1.5874 F=2 T=2^15 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performan ce, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performan ce, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performan ce, raise the target GPU architecture to 75+.
19:36:33 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
19:36:33 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
19:36:33 INFO total_encoding_params=960848 total_network_params=9728
19:36:33 WARNING Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers 19:36:33 WARNING Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed) Got cutlass error: Error Internal at: 346 Could not free memory: /home/yippie/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:465 cudaDeviceSynchronize() failed with error operation not permitted when stream is capturing

$ nvidia-smi
Sun Jul 17 19:44:36 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  On   | 00000000:09:00.0  On |                  N/A |
| 31%   38C    P8    N/A /  75W |    617MiB /  4039MiB |     16%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           On   | 00000000:0C:00.0 Off |                    0 |
| N/A   40C    P8    27W / 149W |      8MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           On   | 00000000:0D:00.0 Off |                    0 |
| N/A   42C    P8    30W / 149W |      8MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

I went ahead and tested with the frequency config (no clue what that is, but it defaults to Cutlass):

19:45:50 INFO     Loading NeRF dataset from
19:45:50 INFO       data/pulseox/transforms.json
19:45:50 SUCCESS  Loaded 98 images after 0s
19:45:50 INFO       cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:45:51 INFO     Loading network config from: configs/nerf/frequency.json
19:45:51 INFO     Loading parent network config from: configs/nerf/base.json
19:45:51 INFO     Density model: 3--[Frequency]-->96--[CutlassMLP(neurons=256,layers=9)]-->1
19:45:51 INFO     Color model:   3--[Frequency]-->24+16--[CutlassMLP(neurons=256,layers=3)]-->3
19:45:51 INFO       total_encoding_params=0 total_network_params=434176
19:45:51 WARNING  Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:45:51 WARNING  Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
19:45:51 ERROR    Uncaught exception: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:567 cuMemCreate(&m_handles.back(), n_bytes_to_allocate, &prop, 0) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:473 cuMemRelease(handle) failed with error CUDA_ERROR_INVALID_VALUE

And when I drop frequency.json's n_neurons to 16, to test:

19:47:08 INFO     Loading NeRF dataset from
19:47:08 INFO       data/pulseox/transforms.json
19:47:08 SUCCESS  Loaded 98 images after 0s
19:47:08 INFO       cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:47:09 INFO     Loading network config from: configs/nerf/frequency.json
19:47:09 INFO     Loading parent network config from: configs/nerf/base.json
19:47:09 INFO     Density model: 3--[Frequency]-->96--[CutlassMLP(neurons=16,layers=9)]-->1
19:47:09 INFO     Color model:   3--[Frequency]-->24+16--[CutlassMLP(neurons=16,layers=3)]-->3
19:47:09 INFO       total_encoding_params=0 total_network_params=4096
19:47:09 WARNING  Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:47:09 WARNING  Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
Got cutlass error: Error Internal at: 346
Could not free memory: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:465 cudaDeviceSynchronize() failed with error operation not permitted when stream is capturing

I have not yet tested reverting to that old revision but do plan to.

jaggzh commented 2 years ago

Update: With an aabb_scale of 2 or 4 in data/pulseox/transforms.json, without a specified config, it does pop up the gui finally:

19:51:45 INFO     Loading NeRF dataset from
19:51:45 INFO       data/pulseox/transforms.json
19:51:45 SUCCESS  Loaded 98 images after 0s
19:51:45 INFO       cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:51:45 INFO     Loading network config from: configs/nerf/base.json
19:51:45 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
19:51:45 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
19:51:45 INFO     Color model:   3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
19:51:45 INFO       total_encoding_params=13074912 total_network_params=9728
19:51:45 WARNING  Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:51:45 WARNING  Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)

With an aabb_scale of 8 it gives the stream is capturing error:

19:54:06 INFO     Loading NeRF dataset from
19:54:06 INFO       data/pulseox/transforms.json
19:54:06 SUCCESS  Loaded 98 images after 0s
19:54:06 INFO       cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:54:07 INFO     Loading network config from: configs/nerf/base.json
19:54:07 INFO     GridEncoding:  Nmin=16 b=1.5874 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
19:54:07 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
19:54:07 INFO     Color model:   3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
19:54:07 INFO       total_encoding_params=13288400 total_network_params=9728
19:54:07 WARNING  Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:54:07 WARNING  Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
Got cutlass error: Error Internal at: 346
Could not free memory: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:465 cudaDeviceSynchronize() failed with error operation not permitted when stream is capturing

jaggzh commented 2 years ago

I'm not closing this since I'm not sure what's going on, or if I'm doing something wrong still, and I'd appreciate suggestions.