search

nerfstudio-project / gsplat

CUDA accelerated rasterization of gaussian splatting
https://docs.gsplat.studio/
Apache License 2.0
1.84k stars 223 forks source link

RuntimeError: CUDA error: no kernel image is available for execution on the device when initializing CUDA for gsplat #302

Open ChaosAdmStudent opened 1 month ago

ChaosAdmStudent commented 1 month ago

I am working with the following:

Ubuntu 22.04.4 LTS Nvidia driver: 555.42.06 Nvidia RTX 3080 Ti

I have multiple cuda installations in the /usr/local folder but I have added the cuda-12.1 bin to PATH and lib64 to LD_LIBRARY_PATH

I have installed the CUDA12.1 pytorch binaries using pip, which based on my online research shows that it ships with the CUDA binaries to run it without external setup.

However, if I don’t add the /usr/local/cuda-12.1/bin to the PATH and try to run the project it throws a “gsplat: No CUDA toolkit found. gsplat will be disabled”. I am assuming this is because this repository has custom CUDA kernels which are not just pure pytorch code.

After adding to PATH, I dont get this error anymore, it starts initializing but fails after a while saying “RuntimeError: CUDA error: no kernel image is available for execution on the device”

Looking at the driver compatibility, it seems that the nvidia driver version 555.42.02+ is not compatible with any other CUDA except 12.5. Is this the reason the code is not running? Do I need a different nvidia driver?

Perhaps this is why I can do basic CUDA device hosted operations in pure pytorch but it fails when trying to run this repository. Any tips would be heavily appreciated as this is all new to me.

ChaosAdmStudent commented 1 month ago

After some tinkering, I figured going with the gsplat installation that compiled gsplat during installation instead of JIT ended up fixing it for me. I wonder why that happened for knowledge sake though!

kstoneriv3 commented 2 weeks ago

I encountered the same error, RuntimeError: CUDA error: no kernel image is available for execution on the device with A30 GPU and Cuda 11.8. In my case, just compiling during the installation did not solve the issue, but I was able to fix it by explicitly setting the compute capability of a GPU by modifying the setup.py as follows, so that it works on any GPUs with compute capability >= 8.0.

    nvcc_flags += ["-gencode=arch=compute_80,code=sm_80", "-gencode=arch=compute_80,code=sm_80"]

So I assume that your error had something to do with targeting the proper compute capability when compiling.

(Edit:) I think a better alternative is to use something like to provide NVCC_FLAGS when installing from source to compile, like

$ NVCC_FLAGS="-gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_80,code=sm_80" pip install git+https://github.com/nerfstudio-project/gsplat.git@v1.0.0