search

NVlabs / instant-ngp

Instant neural graphics primitives: lightning fast NeRF and more
https://nvlabs.github.io/instant-ngp
Other
16k stars 1.93k forks source link

GUI version with "Could not free memory" Error #372

Closed QinlongHuang closed 2 years ago

QinlongHuang commented 2 years ago

I have built the great work NGP with and without GUI. It works fine in no-GUI version. But when I start a GUI version using ./build-gui/testbed --scene=data/nerf/fox I get infos like


10:24:04 INFO       data/nerf/fox/transforms.json
10:24:04 SUCCESS  Loaded 50 images of size 1080x1920 after 0s
10:24:04 INFO       cam_aabb=[min=[1.0229,-1.33309,-0.378748], max=[2.46175,1.00721,1.41295]]
10:24:04 INFO     Loading network config from: configs/nerf/base.json
10:24:04 INFO     GridEncoding:  Nmin=16 b=1.51572 F=2 T=2^19 L=16
10:24:04 INFO     Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
10:24:04 INFO     Color model:   3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
10:24:04 INFO       total_encoding_params=13074912 total_network_params=10240
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:458 cudaDeviceSynchronize() failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:458 cudaDeviceSynchronize() failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:458 cudaDeviceSynchronize() failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:458 cudaDeviceSynchronize() failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:458 cudaDeviceSynchronize() failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:702 cudaEventDestroy(m_training_splitk_events[i]) failed with error unspecified launch failure
/Workspace/instant-ngp/dependencies/tiny-cuda-nn/src/fully_fused_mlp.cu:703 cudaStreamDestroy(m_training_splitk_streams[i]) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:128 cudaFree(rawptr) failed with error unspecified launch failure
10:24:04 ERROR    Uncaught exception: Could not copy to host: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:292 cudaMemcpy(host_data, data(), num_elements * sizeof(T), cudaMemcpyDeviceToHost) failed with error unspecified launch failure
Could not free memory: /Workspace/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:458 cudaDeviceSynchronize() failed with error unspecified launch failure```

System Info
Distro: Ubuntu 16.04
CPU: Intel i9-7920X
GPU: 4 x NVIDIA RTX 2080Ti
I connect the remote linux server without a real monitor plugged in, i.e., I use a virtual display.
So is it a problem of the virtual display ?
muacq commented 2 years ago

Same problem for me. Did you solve this?

Ghul-huan commented 1 year ago

Same problem for me. Did you solve this?