A CUDA error occurred during rendering:an illegal memory access was encountered.

[henrypearce4D]

I successfully trained the coffee_martini dataset but get this error when trying to view it on the pre-build windows viewer;

C:\Users\INFINITE>D:/.repos/spacetime/viewer/bin/SIBR_gaussianViewer_app_rwdi.exe -m  D:\.repos\spacetime\output\coffee_martini\test
[SIBR] --  INFOS  --:    main process here
[SIBR] --  INFOS  --:   Initialization of GLFW
[SIBR] --  INFOS  --:   OpenGL Version: 4.6.0 NVIDIA 528.49[major: 4, minor: 6]
[SIBR] ##  ERROR  ##:   FILE C:\Users\S9053246\code\gaussian-splatting-release-poly-lite\SIBR_viewers_poly_lite\src\core\scene\ParseData.cpp
                        LINE 560, FUNC sibr::ParseData::getParsedData
                        Cannot determine type of dataset at /data/coffee_martini/colmap_0
[SIBR] --  INFOS  --:   Did not find specified input folder, loading from model path
Number of input Images to read: 900
Number of Cameras set up: 900
[SIBR] --  INFOS  --:   Mesh contains: colors: 1, normals: 1, texcoords: 1
[SIBR] --  INFOS  --:   Mesh 'D:\.repos\spacetime\output\coffee_martini\test/input.ply successfully loaded. 1 meshes were loaded with a total of  (0) faces and  (351385) vertices detected. Init GL ...
[SIBR] --  INFOS  --:   Init GL mesh complete
[SIBR] --  INFOS  --:   Loading 251037 Gaussian splats
[SIBR] --  INFOS  --:   Loading done!
[SIBR] --  INFOS  --:   Initializing Raycaster
[SIBR] --  INFOS  --:   Interactive camera using (0.009,1100) near/far planes.
Switched to trackball mode.
[SIBR] ##  ERROR  ##:   FILE C:\Users\S9053246\code\gaussian-splatting-release-poly-lite\SIBR_viewers_poly_lite\src\projects\gaussianviewer\renderer\GaussianView.cpp
                        LINE 716, FUNC sibr::GaussianView::onRenderIBR
                        A CUDA error occurred during rendering:an illegal memory access was encountered. Please rerun in Debug to find the exact line!

I can view the n3d_sear_steak_lite_allcam no problem. This dataset has windows, and there is alot of points outside, could this be the cause of the error?

[lizhan17]

The viewer currently only support light version without guided sampling.

for some scenes contain the guided sampling, the viewer will not be compatible to the guided sampling after you see the scene through the viewer, throwing a cuda memory access error. I am working on the viewer. You can select the other scenes or set "--emsstart 30000" to disable the guided sampling to view the scene through the viewer. The test script should work with guided sampling anyway.

[henrypearce4D]

thanks for the explanation, is --emsstart 30000 a viewer command? i still get the CUDA error if I add it.

Also additionally, another issue I found is

(feature_splatting) infinite@nerf-linux:~/SpacetimeGaussians$ python train.py --quiet --eval --config configs/n3d_lite/sear_steak.json --source_path data/training/sear_steak/colmap_0  --model_path data/output/sear_steak
Optimizing data/output/sear_steak
Traceback (most recent call last):
  File "train.py", line 392, in <module>
  File "train.py", line 69, in train
  File "/home/infinite/SpacetimeGaussians/thirdparty/gaussian_splatting/scene/__init__.py", line 50, in __init__
  File "./thirdparty/gaussian_splatting/scene/dataset_readers.py", line 787, in readColmapSceneInfo
  File "./thirdparty/gaussian_splatting/scene/dataset_readers.py", line 164, in readColmapCameras
  File "/home/infinite/anaconda3/envs/feature_splatting/lib/python3.7/site-packages/PIL/Image.py", line 3227, in open
OSError: [Errno 24] Too many open files: '/home/infinite/SpacetimeGaussians/data/training/sear_steak/colmap_14/images/cam00.png'

setting the config to 15 frames it trains, the coffee_martini trans fine at 50

[lizhan17]

OSError: [Errno 24] Too many open files can be solved by first executing "ulimit -n 4096" in terminal before you run the training code.

--emsstart 30000 is the command to train the model without guided sampling.

[szat]

Have the same issue, any tips?

[lizhan17]

Have the same issue, any tips?

which error ? for the viewer or training process?

[szat]

hi, it is with the viewer, the training seems to finish fine.

[lizhan17]

hi, it is with the viewer, the training seems to finish fine.

are you using full model? the viewer cannot work with full model.

We also uploaded the latest viewer on hugging face 4 months ago. Please select the viewer based on your cuda driver version.

[szat]

Ok the compilation of the viewer also worked for me, now it displays but i am getting intermittent errors, i suppose it is a memory issue. I have to manually replace file by file for the cudarasterizer, and for the project file i could do it in one go.

Moreover I was using the conda env from 3DGS, then installed the packages as shown in the script manually.

[lizhan17]

intermittent errors like "A CUDA error occurred during rendering:an illegal memory access was encountered."

do you have large enough gpu memory ? for the released demo model, we prune the number of poinst a lot. I guess if you train a model with many points, the gpu memory is an issue.

in that case. reduce rensifcation and disable guided sampling.

"densify": 3, "desicnt": 2, "emsstart": 30000