RuntimeError: cudaErrorUnknown(999): unknown error

[WANGRUI-ZB]

I run according to the readme documentation this step：Running 3DGS+T runevc -t gui -c configs/exps/gaussiant/gaussiant_${expname}.yaml,configs/specs/superm.yaml encounter RuntimeError: cudaErrorUnknown(999): unknown error Please help me.

[dendenxu]

Hi @WANGRUI-ZB sorry for the late reply. What PyTorch and CUDA version are you using? I'll try to reproduce. It's best if you could paste the output of pip list here for me to try to recreate the environment.

Moreover, could you please also check if the display variable is set in your terminal? echo $DISPLAY

[WANGRUI-ZB]

Hi @dendenxu thank you for your reply. My PyTorch version is 2.1.1 CUDA version is 12.3. and I check the display variable with echo $DISPLAY print :0 The following is the result of pip list:

absl-py                   2.0.0
addict                    2.4.0
ansi2html                 1.9.1
antlr4-python3-runtime    4.9.3
anyio                     4.1.0
appdirs                   1.4.4
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-lru                 2.0.4
attrs                     23.1.0
autopep8                  2.0.4
Babel                     2.13.1
beautifulsoup4            4.12.2
black                     23.11.0
bleach                    6.1.0
blessed                   1.20.0
blinker                   1.7.0
Brotli                    1.1.0
cachetools                5.3.2
calmsize                  0.1.3
certifi                   2023.11.17
cffi                      1.16.0
charset-normalizer        3.3.2
chumpy                    0.70
clang-format              17.0.6
click                     8.1.7
cloudpickle               3.0.0
cmake                     3.27.2
comm                      0.2.0
commentjson               0.9.0
ConfigArgParse            1.7
contourpy                 1.2.0
cuda-python               12.3.0
cycler                    0.12.1
dash                      2.14.2
dash-core-components      2.0.0
dash-html-components      2.0.0
dash-table                5.0.0
debugpy                   1.8.0
decorator                 5.1.1
defusedxml                0.7.1
detectron2                0.6
diff-gauss                1.0.5
docstring-parser          0.15
dotmap                    1.3.30
easyvolcap                0.0.0              /data/service/easyvolcap-public
einops                    0.7.0
et-xmlfile                1.1.0
exceptiongroup            1.2.0
executing                 2.0.1
fast-autocomplete         0.9.0
fastjsonschema            2.19.0
filelock                  3.12.2
Flask                     3.0.0
flatbuffers               23.5.26
fonttools                 4.46.0
fqdn                      1.5.1
fsspec                    2023.12.2
func-timeout              4.3.5
fvcore                    0.1.5.post20221221
glfw                      2.6.3
gmpy2                     2.1.2
google-auth               2.25.2
google-auth-oauthlib      1.1.0
gpustat                   1.2.dev19+gf4cd639
grpcio                    1.60.0
h5py                      3.10.0
human-body-prior          2.2.2.0
hydra-core                1.3.2
idna                      3.6
imageio                   2.33.1
imgui-bundle              1.1.0
importlib-metadata        7.0.0
iniconfig                 2.0.0
iopath                    0.1.9
ipdb                      0.13.13
ipykernel                 6.27.1
ipython                   8.18.1
ipywidgets                8.1.1
isoduration               20.11.0
itsdangerous              2.1.2
jedi                      0.19.1
Jinja2                    3.1.2
joblib                    1.3.2
json5                     0.9.14
jsonpointer               2.4
jsonschema                4.20.0
jsonschema-specifications 2023.11.2
jupyter                   1.0.0
jupyter_client            8.6.0
jupyter-console           6.6.3
jupyter_core              5.5.0
jupyter-events            0.9.0
jupyter-lsp               2.2.1
jupyter_server            2.12.1
jupyter_server_terminals  0.5.0
jupyterlab                4.0.9
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.2
jupyterlab-widgets        3.0.9
kiwisolver                1.4.5
kornia                    0.7.0
lark-parser               0.7.8
lazy_loader               0.3
lit                       16.0.6
lpips                     0.1.4
Mako                      1.3.0
Markdown                  3.5.1
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.8.2
matplotlib-inline         0.1.6
mdurl                     0.1.2
mediapipe                 0.10.8
memory-tempfile           2.2.3
mistune                   3.0.2
mpmath                    1.3.0
msgpack                   1.0.7
munch                     4.0.0
mypy-extensions           1.0.0
nbclient                  0.9.0
nbconvert                 7.12.0
nbformat                  5.7.0
nest-asyncio              1.5.8
networkx                  3.1
ninja                     1.11.1.1
notebook                  7.0.6
notebook_shim             0.2.3
numpy                     1.25.2
nvdiffrast                0.3.1
nvidia-ml-py              12.535.133
oauthlib                  3.2.2
olefile                   0.47
omegaconf                 2.3.0
open3d                    0.17.0
opencv-contrib-python     4.8.1.78
opencv-python             4.8.1.78
openpyxl                  3.1.2
overrides                 7.4.0
packaging                 23.2
pandas                    2.1.4
pandocfilters             1.5.0
parso                     0.8.3
pathspec                  0.12.1
pdbr                      0.8.8
pexpect                   4.9.0
Pillow                    10.1.0
pip                       23.3.1
platformdirs              4.1.0
plotly                    5.18.0
pluggy                    1.3.0
plyfile                   1.0.2
portalocker               2.8.2
prometheus-client         0.19.0
prompt-toolkit            3.0.41
protobuf                  3.20.3
psutil                    5.9.6
ptyprocess                0.7.0
pure-eval                 0.2.2
py-cpuinfo                9.0.0
pyasn1                    0.5.1
pyasn1-modules            0.3.0
pycocotools               2.0.7
pycodestyle               2.11.1
pycparser                 2.21
pycuda                    2023.1
PyGLM                     2.7.1
Pygments                  2.17.2
PyMCubes                  0.1.4
pymeshlab                 2022.2.post4
pyntcloud                 0.3.1
PyOpenGL                  3.1.7
PyOpenGL-accelerate       3.1.7
pyparsing                 3.1.1
pyperclip                 1.8.2
pyquaternion              0.9.9
PySocks                   1.7.1
pytest                    7.4.3
pytest-benchmark          4.0.0
python-dateutil           2.8.2
python-json-logger        2.0.7
pytools                   2023.1.1
pytorch-memlab            0.3.0
pytorch-msssim            1.0.0
pytorch3d                 0.7.5
PyTurboJPEG               1.7.3
pytz                      2023.3.post1
PyYAML                    6.0.1
pyzmq                     25.1.2
qtconsole                 5.5.1
QtPy                      2.4.1
referencing               0.32.0
requests                  2.31.0
requests-oauthlib         1.3.1
retrying                  1.3.4
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rich                      13.7.0
rpds-py                   0.13.2
rsa                       4.9
ruamel.yaml               0.18.5
ruamel.yaml.clib          0.2.8
scikit-image              0.22.0
scikit-learn              1.3.2
scipy                     1.11.4
Send2Trash                1.8.2
setuptools                68.0.0
shtab                     1.6.5
simple-knn                0.0.0
six                       1.16.0
smplx                     0.1.28
sniffio                   1.3.0
sounddevice               0.4.6
soupsieve                 2.5
stack-data                0.6.3
sympy                     1.12
tabulate                  0.9.0
tenacity                  8.2.3
tensorboard               2.15.1
tensorboard-data-server   0.7.2
tensorboardX              2.6.2.2
termcolor                 2.4.0
terminado                 0.18.0
threadpoolctl             3.2.0
tifffile                  2023.12.9
timg                      1.1.6
tinycss2                  1.2.1
tinycudann                1.7
tomli                     2.0.1
torch                     2.1.1
torch-scatter             2.1.2
torch-tb-profiler         0.4.3
torchaudio                2.1.1
torchdiffeq               0.2.3
torchmcubes               0.1.0
torchvision               0.16.1
tornado                   6.4
tqdm                      4.66.1
traitlets                 5.14.0
trimesh                   4.0.5
triton                    2.1.0
types-python-dateutil     2.8.19.14
typing_extensions         4.7.1
tyro                      0.6.0
tzdata                    2023.3
ujson                     5.9.0
uri-template              1.3.0
urllib3                   2.1.0
wcwidth                   0.2.12
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.7.0
Werkzeug                  3.0.1
wheel                     0.41.2
widgetsnbextension        4.0.9
xatlas                    0.0.8
yacs                      0.1.8
yapf                      0.40.2
zipp                      3.17.0

[dendenxu]

Hi, looks like you're using the most cutting-edge CUDA and nvidia driver, which we haven't tested. I'll try to setup a test environment for testing and get back to you later.

In the meantime, you could try creating a new environment with slighter lower CUDA version (we've tested 12.1 and 11.8) and matched pytorch installation.

[WANGRUI-ZB]

Hi, can you tell me the recommended cuda and pytorch version? before I tried cuda 11.8 but report the same error!

[dendenxu]

PyTorch 1.12.1 -> 2.0.1 and CUDA 11.8 -> 12.1 have been tested on Ubuntu and Windows. Also, is your testing environment a double-GPU laptop? I suspect the error could be due to mismatches between the OpenGL's context GPU and CUDA's GPU.

[dendenxu]

I suspect the error could be due to mismatches between the OpenGL's context GPU and CUDA's GPU.

If that's the case, you could try disabling the integrated graphics card in BIOS. For now, easyvolcap only supports Nvidia GPUs. But I think it shouldn't be too hard to skip creating OpenGL context on integrated intel graphics cards in the future.

[WANGRUI-ZB]

Okay, I'll try to close it and run

[dendenxu]

Hi @WANGRUI-ZB. Did the provided solution solve your issue? Do you still face the same exception after disabling the integrated graphics card? I'll be happy to provide more assistance if the issue persists : ]

[WANGRUI-ZB]

I have been busy with other things at this time, so I have not yet operated. I will give you the feedback when I get the result

[WANGRUI-ZB]

I've verified that it works with the integrated graphics card disabled

[dendenxu]

@WANGRUI-ZB Glad to know! Closing the issue as completed for now. Will reopen if after we added integrated graphics support.

[Linkersem]

hi, @dendenxu i have also meet similar problem ,and my env is this, what should i do to solve this?

Package Version Editable project location

---

absl-py 2.1.0 addict 2.4.0 asttokens 2.4.1 av 12.0.0 boltons 23.0.0 brotlipy 0.7.0 certifi 2022.12.7 cffi 1.15.1 charset-normalizer 2.0.4 colorama 0.4.6 conda 23.3.1 conda-content-trust 0.1.3 conda-package-handling 2.0.2 conda_package_streaming 0.7.0 contourpy 1.2.1 cryptography 39.0.1 cuda-python 12.4.0 cycler 0.12.1 decorator 5.1.1 diff-gaussian-rasterization 0.0.0 diff-gaussian-rasterization-wodilate 0.0.0 easyvolcap 0.0.0 /mnt/data/guoyl16/project/LoG/submodules/EasyVolcap einops 0.8.0 exceptiongroup 1.2.1 executing 2.0.1 filelock 3.13.1 fonttools 4.51.0 fsspec 2024.2.0 fvcore 0.1.5.post20221221 glfw 2.7.0 grpcio 1.63.0 h5py 3.11.0 huggingface-hub 0.23.0 icecream 2.1.3 idna 3.4 imageio 2.34.1 imgui-bundle 1.3.0 importlib_metadata 7.1.0 imutils 0.5.4 install 1.3.5 iopath 0.1.10 ipdb 0.13.13 ipython 8.24.0 jedi 0.19.1 Jinja2 3.1.3 jsonpatch 1.32 jsonpointer 2.1 kiwisolver 1.4.5 lazy_loader 0.4 LoG 0.0 /mnt/data/guoyl16/project/LoG lpips 0.1.4 Markdown 3.6 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.9.0 matplotlib-inline 0.1.7 mdurl 0.1.2 mpmath 1.3.0 munch 4.0.0 networkx 3.2.1 numpy 1.26.4 nvidia-cublas-cu11 11.11.3.6 nvidia-cuda-cupti-cu11 11.8.87 nvidia-cuda-nvrtc-cu11 11.8.89 nvidia-cuda-runtime-cu11 11.8.89 nvidia-cudnn-cu11 8.7.0.84 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.3.0.86 nvidia-cusolver-cu11 11.4.1.48 nvidia-cusparse-cu11 11.7.5.86 nvidia-nccl-cu11 2.19.3 nvidia-nvtx-cu11 11.8.86 opencv-python 4.9.0.80 packaging 23.0 pandas 2.2.2 parso 0.8.4 pdbr 0.8.8 pexpect 4.9.0 pillow 10.2.0 pip 23.0.1 platformdirs 4.2.2 pluggy 1.0.0 plyfile 1.0.3 portalocker 2.8.2 prompt-toolkit 3.0.43 protobuf 5.26.1 ptyprocess 0.7.0 pure-eval 0.2.2 pycosat 0.6.4 pycparser 2.21 PyGLM 2.7.1 Pygments 2.18.0 PyMCubes 0.1.4 pyntcloud 0.3.1 PyOpenGL 3.1.7 pyOpenSSL 23.0.0 pyparsing 3.1.2 pyperclip 1.8.2 PySocks 1.7.1 python-dateutil 2.9.0.post0 pytorch3d 0.7.6 PyTurboJPEG 1.7.3 pytz 2024.1 PyYAML 6.0.1 requests 2.28.1 rich 13.7.1 ruamel.yaml 0.17.21 ruamel.yaml.clib 0.2.6 scikit-image 0.23.2 scipy 1.13.0 setuptools 65.6.3 simple-knn 0.0.0 six 1.16.0 stack-data 0.6.3 sympy 1.12 tabulate 0.9.0 tensorboard 2.16.2 tensorboard-data-server 0.7.2 tensorboardX 2.6.2.2 termcolor 2.4.0 tifffile 2024.5.10 timm 0.6.12 tomli 2.0.1 toolz 0.12.0 torch 2.2.0+cu118 torchaudio 2.2.0+cu118 torchvision 0.17.0+cu118 tqdm 4.65.0 traitlets 5.14.3 trimesh 4.4.0 triton 2.2.0 typing_extensions 4.9.0 tzdata 2024.1 ujson 5.10.0 urllib3 1.26.15 wcwidth 0.2.13 websockets 12.0 Werkzeug 3.0.3 wheel 0.38.4 yacs 0.1.8 yapf 0.40.2 zipp 3.18.2 zstandard 0.19.0

[dendenxu]

Hi, sorry for the late reply. It seems like this is an issue with the downstream code repo LoG, maybe we should continue the discussion there. Looks like the issue originates from a wrongly installed LoG environment for gaussian rasterization.