nerfstudio-project / nerfacc

A General NeRF Acceleration Toolbox in PyTorch.
https://www.nerfacc.com/
Other
1.37k stars 113 forks source link

train_ngp_nerf_occ.py: RuntimeError: CUDA error: invalid configuration argument #207

Open sararoma95 opened 1 year ago

sararoma95 commented 1 year ago

Hello,

It may be a easy problem to solve but I have not be able to do it.

When running python examples/train_ngp_nerf_occ.py --scene lego --data_root path I get the following error when I evaluate the model (L. 236):

RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

This does not happen when using train_ngp_nerf_prop.py

Thanks!

liruilong940607 commented 1 year ago

This is weird. May I know your CUDA and pytorch version?

python -c "import torch; print(torch.__version__)"
nvcc --version

And what's your GPU?

nvidia-smi

My guess this is related to gpu you are using.

aiyb1314 commented 1 year ago

Hello, I have the same problem! CUDA and pytorch version: 2.0.0+cu118, Cuda compilation tools, release 11.8 GPU:12.0

liruilong940607 commented 1 year ago

Sorry I mean which NVIDIA card are you using, e.g. V100?

sararoma95 commented 1 year ago

Hello I’m using A100 thanks Regards,Sara RojasOn 25 Apr 2023, at 10:24 PM, Ruilong Li @.***> wrote: Sorry I mean which NVIDIA card are you using, e.g. V100?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

aiyb1314 commented 1 year ago

RTX 3090

filipgu commented 1 year ago

also having problems on A100 with occupancy grids. Rays are in bounding box, but ray indices, starts and ends come back with 0 in the batch dimension.

mJones00 commented 1 year ago

I also had this exact issue on my 3090, although it was working fine in another conda environment. Turns out that when I set up the new environment, I forgot to specify the torch version and therefore I was using torch 2.0.0 with cuda 11.7, then tiny-cuda-nn was compiled with this combination. Installing torch version 1.13.0 with pytorch-cuda 11.7 fixed the issue for me.

liruilong940607 commented 1 year ago

I can reproduce this error with torch 2.0.0. With torch 1.13.0 everything seems working fine.

I'll come back to this issue once I figure out why this happens. In the mean time, using torch 1.13.0 seems to be a workaround.

sararoma95 commented 1 year ago

Thanks!

On Tue, May 2, 2023 at 3:10 AM Ruilong Li(李瑞龙) @.***> wrote:

I can reproduce this error with torch 2.0.0. With torch 1.13.0 everything seems working fine.

I'll come back to this issue once I figure out why this happens. In the mean time, using torch 1.13.0 seems to be a workaround.

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/KAIR-BAIR/nerfacc/issues/207*issuecomment-1530635985__;Iw!!Nmw4Hv0!yg9uH9frG5qkdhyd16cc_UdzvfZsk9eZtN20yBug2ojRTIRroLs1cpcMxHBZop57a0_YtJbPANmU7j3eLMd_euT0o1fH3HPVEYQ$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AWZC2PIOSBGVOQ3GEUT5Q2TXEBGG5ANCNFSM6AAAAAAXJDPSLQ__;!!Nmw4Hv0!yg9uH9frG5qkdhyd16cc_UdzvfZsk9eZtN20yBug2ojRTIRroLs1cpcMxHBZop57a0_YtJbPANmU7j3eLMd_euT0o1fHgCjw0B4$ . You are receiving this because you authored the thread.Message ID: @.***>

--

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

liruilong940607 commented 1 year ago

Just fixed it on the master branch!

1z1y commented 1 year ago

I also have this problem with torch1.10 and cuda 11.3

AIBluefisher commented 1 year ago

I have this problem with torch 1.11 and cuda 11.3 while I use NVIDIA 4090. Tracing stacks:

  File "/home/xxx/anaconda3/envs/conerf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/media/.../model/render_utils.py", line 772, in render_sdf_image_with_xxx
    weights, _ = render_weight_from_alpha(
  File "/home/xxx/anaconda3/envs/conerf/lib/python3.9/site-packages/nerfacc/volrend.py", line 305, in render_weight_from_alpha
    trans = render_transmittance_from_alpha(
  File "/home/xxx/anaconda3/envs/conerf/lib/python3.9/site-packages/nerfacc/volrend.py", line 201, in render_transmittance_from_alpha
    packed_info = pack_info(ray_indices, n_rays)
  File "/home/xxx/anaconda3/envs/conerf/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/xxx/anaconda3/envs/conerf/lib/python3.9/site-packages/nerfacc/pack.py", line 43, in pack_info
    chunk_cnts = torch.zeros((n_rays,), device=device, dtype=dtype)
RuntimeError: CUDA error: invalid configuration argument
AIBluefisher commented 1 year ago

I'm here to contribute more data points. I encountered the same issue with torch 1.11 and cuda 11.3 while I use NVIDIA 4090. The issue always happens for the render_image() function during validation. Different versions of nerfacc I have tried so far: 0.3.2, 0.3.3, 0.3.4, 0.3.5.

ahmadki commented 11 months ago

I've encountered the same problem, and after two days of debugging, I believe I've figured it out.

The error is not related to the GPU model nor the CUDA version. This is not even a NerfAcc bug.

The error message that I see is similar to the one reported by @AIBluefisher:

File "/home/xxx/anaconda3/envs/conerf/lib/python3.9/site-packages/nerfacc/pack.py", line 43, in pack_info
    chunk_cnts = torch.zeros((n_rays,), device=device, dtype=dtype)
RuntimeError: CUDA error: invalid configuration argument

But CUDA entered an invalid state before this point, and torch.zeros is likely the first CUDA kernel being called post-error. Unfortunately, setting CUDA_LAUNCH_BLOCKING=1 doesn't help to pinpoint the source of the issue.

I added torch.zeros((1000,), device="cuda", dtype=torch.int64) in different parts of my code to trace back to the function that was causing CUDA to malfunction. In my case, it was TorchNGP GridEncoder, specifically when the GridEncoder is invoked with an empty positions tensor (positions.shape=[0, 3]).

The positions tensor is empty because OccGridEstimator was being called before training and before any calls to update_every_n_steps, which results in all zero binary values (in other words, OccGridEstimator skips all rays). This is particularly common with pytorch lightning, where the framework performs sanity check iterations before training.

As a fix, I simply bypassed the NeRF / TorchNGP when the input positions are empty:

if positions.shape[0] == 0:
    sigma = torch.zeros(0, device=positions.device)
    color = torch.zeros(0, 3, device=positions.device)
    return sigma, color

Alternatively, it might be worth explore calling update_every_n_steps right after initializing the OccGridEstimator.

weihan1 commented 11 months ago

Hi, I also encountered this issue:

File "/miniconda3/envs/multiview/lib/python3.8/site-packages/nerfacc/vol_rendering.py", line 338, in render_visibility visibility, packed_info_visible = _C.rendering_alphas_forward( File "miniconda3/envs/multiview/lib/python3.8/site-packages/nerfacc/cuda/__init__.py", line 13, in call_cuda return getattr(_C, name)(*args, **kwargs) RuntimeError: CUDA error: invalid configuration argument

I'm using Pytorch 1.12.1+cuda11.6 and CUDA compilation tools release 11.4. Also, using NVIDIA RTX A6000

Rlee719 commented 9 months ago

Same problem in version 0.5.2, volrend.accumulate_along_rays can not handle empty input

stevehan00 commented 8 months ago

I faced the same issue using torch==1.12.1+cuda11.3 in RTX 4090, but it could be solved by replacing the version with torch 1.13.1+cuda11.6

mashad98 commented 3 months ago

anyone an idea how i can solve this?? i am not a developer but a designer using SVD..