RightHandAxis
commented
3 months ago After a few days of debugging, I've managed to fix the problem eventually!!
In my case, i have to make sure that the cudatoolkit version is 11.8 to compile the submodules/diff-gaussian-rasterization and add "-Xcompiler -fno-gnu-unique" option in submodules/diff-gaussian-rasterization/setup.py: line 29 resolves the "cuda illegal memory access error" and "RuntimeError: numel: integer multiplication overflow" in training. Just do as recommended in this issue https://github.com/graphdeco-inria/gaussian-splatting/issues/41
... 29 extra_compile_args={"nvcc": ["-Xcompiler", "-fno-gnu-unique","-I" + os.path.join(os.path.dirname(os.path.abspath(__file__)), "third_party/glm/")]}) ...
In addition, if your server or cluster has multiple CUDA versions installed and you do not want to mess them up, I recommend you to install cudatoolkit=11.8 as it is recommended in this issue https://github.com/TimDettmers/bitsandbytes/issues/212)
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
The above command line will install cudatoolkit via conda completely including CUDA compiler (e.g., "nvcc").
Hope this solution could help with others.
Hi authors, thanks a lot for the excellent work and I appreciate for the public codes ! As the title shows, it runs into the problem when the algorithm passes the diff_gaussian_rasterization, especially the following line: num_rendered, color, depth, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args) in the "diff_gaussian_rasterization/init.py " file.
And I notice that someone has already mentioned this issue (no solution provided), but I can not find a solution. Is there any workaround to fix the problem? Could you please help with that?
Thanks in advance!!