search

a-lemus96 / fs-nerf

PyTorch implementation for experimenting with frequency regularized Neural Radiance Fields.
0 stars 0 forks source link

Solve `CUDA INTERNAL ASSET FAILED` error #67

Closed a-lemus96 closed 3 months ago

a-lemus96 commented 3 months ago

Solve CUDA INTERNAL ASSET FAILED error

Traceback (most recent call last):
  File "/home/lemus/projects/fs-nerf/src/run-nerf.py", line 534, in <module>
    main()
  File "/home/lemus/projects/fs-nerf/src/run-nerf.py", line 434, in main
    train(
  File "/home/lemus/projects/fs-nerf/src/run-nerf.py", line 234, in train
    render_output = renderer.render_rays(rays_o, rays_d, model)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lemus/projects/fs-nerf/src/render/renderer.py", line 82, in render_rays
    ray_idxs, t_starts, t_ends = self.estimator.sampling(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/nerfacc/estimators/occ_grid.py", line 164, in sampling
    intervals, samples, _ = traverse_grids(
                            ^^^^^^^^^^^^^^^
  File "/home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/nerfacc/grid.py", line 158, in traverse_grids
    t_mins, t_maxs, hits = ray_aabb_intersect(rays_o, rays_d, aabbs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/nerfacc/grid.py", line 43, in ray_aabb_intersect
    t_mins, t_maxs, hits = _C.ray_aabb_intersect(
                           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/nerfacc/cuda/__init__.py", line 13, in call_cuda
    return getattr(_C, name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED at "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/torch/include/c10/cuda/impl/CUDAGuardImpl.h":25, please report a bug to PyTorch.

Originally posted by @a-lemus96 in #61

a-lemus96 commented 3 months ago

Problem occurred when running python run-nerf.py --debug command.

a-lemus96 commented 3 months ago

Torch version currently being used is 

>>> torch.__version__
'2.0.1+cu117'
a-lemus96 commented 3 months ago

By doing some initial research, I found that problem might be due to a possible torch.tensor() declaration that is not being transferred to GPU, according to this issue

PyTorch RuntimeError t == DeviceType::CUDAINTERNAL ASSERT FAILED

a-lemus96 commented 3 months ago

Mismatch between rays_o device and renderer device

Loading model from: /home/lemus/miniconda3/envs/nerf/lib/python3.11/site-packages/lpips/weights/v0.1/vgg.pth
[NeRF]:   0%|                                                                                                                      | 0/8000 [00:00<?, ?it/s]> /home/lemus/projects/fs-nerf/src/render/renderer.py(83)render_rays()
-> ray_idxs, t_starts, t_ends = self.estimator.sampling(
(Pdb) p rays_o.device
device(type='cpu')
(Pdb)
a-lemus96 commented 3 months ago

Problem solved by properly moving tensors to CUDA device.