cuda error

[aiyb1314]

RuntimeError: Cuda error: 9[cudaLaunchKernel(func_tbl[func_idx], gridSize, blockSize, args, 0, stream);]

[aiyb1314]

Traceback (most recent call last): File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/xuj/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in cli.main() File "/home/xuj/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main run() File "/home/xuj/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file runpy.run_path(target, run_name="main") File "/home/xuj/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path return _run_module_code(code, init_globals, run_name, File "/home/xuj/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code _run_code(code, mod_globals, init_globals, File "/home/xuj/.vscode-server/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "/home/xuj/trimip/main.py", line 102, in main() File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper utils.augment_exception_message_and_reraise(e, err_str) File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise raise proxy.with_traceback(exception.traceback) from None File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper return fn(*new_args, new_kwargs) File "/home/xuj/trimip/main.py", line 62, in main trainer.fit() File "/home/xuj/trimip/trainer/trainer.py", line 114, in fit metrics = self.train_iter( File "/home/xuj/trimip/trainer/trainer.py", line 71, in train_iter rb = self.model(cam_rays, target.render_bkgd) File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/xuj/trimip/neural_field/model/trimipRF.py", line 87, in forward ray_indices, t_starts, t_ends = nerfacc.ray_marching( File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/nerfacc/ray_marching.py", line 196, in ray_marching sigmas = sigma_fn(t_starts, t_ends, ray_indices) File "/home/xuj/trimip/neural_field/model/trimipRF.py", line 85, in sigma_fn return self.field.query_density(positions, level_vol)['density'] File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/home/xuj/trimip/neural_field/field/trimipRF.py", line 127, in query_density enc = self.encoding( File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/xuj/trimip/neural_field/encoding/tri_mip.py", line 55, in forward enc = nvdiffrast.torch.texture( File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/nvdiffrast/torch/ops.py", line 613, in texture return _texture_func_mip.apply(filter_mode, tex, uv, uv_da, mip_level_bias, mip_wrapper, filter_mode_enum, boundary_mode_enum, mip_stack) File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply return super().apply(args, kwargs) # type: ignore[misc] File "/home/xuj/anaconda3/envs/zipnerf/lib/python3.10/site-packages/nvdiffrast/torch/ops.py", line 484, in forward out = _get_plugin().texture_fwd_mip(tex, uv, uv_da, mip_level_bias, mip_wrapper, mip_stack, filter_mode_enum, boundary_mode_enum) RuntimeError: Cuda error: 9[cudaLaunchKernel(func_tbl[func_idx], gridSize, blockSize, args, 0, stream);] In call to configurable 'main' (<function main at 0x7ff7ae440280>)

[Lingyun0109]

一样的问题请问你解决了吗 : )

[wbhu]

It seems related to nvdiffrast and GPU type when too many queries are fed into it. I will release a CUDA implementation in the near future

[Lingyun0109]

It seems related to nvdiffrast and GPU type when too many queries are fed into it. I will release a CUDA implementation in the near future

Thanks♪(･ω･)ﾉ！