nerfstudio-project / nerfacc

A General NeRF Acceleration Toolbox in PyTorch.
https://www.nerfacc.com/
Other
1.38k stars 113 forks source link

Test Error #235

Open aiyb1314 opened 1 year ago

aiyb1314 commented 1 year ago

RuntimeError: CUDA error: invalid configuration argument CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

liruilong940607 commented 1 year ago

Hi it is hard to tell what could be wrong from this among of information. One possible trigger of this might be zero shape tensor being sent to tiny-cuda-nn.

sweeneychris commented 1 year ago

I hit this as well when running train_ngp_nerf_occ.py. It always occurs after training so my suspicion is that render_image_with_occgrid_test is somehow the offender. I tested this using the nerf synthetic datasets

aiyb1314 commented 11 months ago

During the model testing phase, the following error occurs : File "/home/xuj/anaconda3/envs/nerfacc/lib/python3.10/site-packages/tinycudann-1.7-py3.10-linux-x86_64.egg/tinycudann/modules.py", line 89, in forward native_ctx, output = native_tcnn_module.fwd(input, params) RuntimeError: /home/xuj/tiny-cuda-nn/include/tiny-cuda-nn/cutlass_matmul.h:332 status failed with error Error Internal After debugging, it is found that it is caused by the number of sampling points in the test phase is 0. How to avoid this phenomenon?

Luh1124 commented 1 month ago

I encountered the same issue, during testing traverse_grids returned an empty [].