search

ShijieZhou-UCLA / feature-3dgs

[CVPR 2024 Highlight] Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields
Other
381 stars 25 forks source link

An error occured during training "RuntimeError: an illegal memory access was encountered" #22

Open zrporz opened 6 months ago

zrporz commented 6 months ago

I have a similar problem like https://github.com/ShijieZhou-UCLA/feature-3dgs/issues/14 , I found that this error occurred once I open --speedup . Any suggestions? 

Optimizing output/0530_figurines
Output folder: output/0530_figurines [30/05 13:10:40]
Reading camera 299/299 [30/05 13:10:41]
Loading Training Cameras [30/05 13:10:41]
Loading Test Cameras [30/05 13:10:48]
Number of points at initialisation :  50517 [30/05 13:10:48]
Training progress:   0%|                                                                                   | 0/30000 [00:00<?, ?it/s]
[CUDA ERROR] in cuda_rasterizer/rasterizer_impl.cu
Line 333: an illegal memory access was encountered
An error occured in forward. Please forward snapshot_fw.dump for debugging. [30/05 13:10:48]
Traceback (most recent call last):
  File "train.py", line 257, in <module>
    training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
  File "train.py", line 105, in training
    render_pkg = render(viewpoint_cam, gaussians, pipe, background)
  File "/datadrive/zhourenping/feature-3dgs/gaussian_renderer/__init__.py", line 256, in render
    var_loss = var_loss)
  File "/datadrive/zhourenping/miniconda3/envs/feature_3dgs/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/datadrive/zhourenping/miniconda3/envs/feature_3dgs/lib/python3.7/site-packages/diff_gaussian_rasterization/__init__.py", line 249, in forward
    raster_settings,
  File "/datadrive/zhourenping/miniconda3/envs/feature_3dgs/lib/python3.7/site-packages/diff_gaussian_rasterization/__init__.py", line 45, in rasterize_gaussians
    raster_settings,
  File "/datadrive/zhourenping/miniconda3/envs/feature_3dgs/lib/python3.7/site-packages/diff_gaussian_rasterization/__init__.py", line 102, in forward
    raise ex
  File "/datadrive/zhourenping/miniconda3/envs/feature_3dgs/lib/python3.7/site-packages/diff_gaussian_rasterization/__init__.py", line 98, in forward
    num_rendered, color, feature_map, depth, num_gauss, mode_id, modes, accum_alpha, radii, point_list, means2D, conic_opacity, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args) ###d
RuntimeError: an illegal memory access was encountered
branyang02 commented 5 months ago

Same issue.

yzslab commented 5 months ago

Try this: https://github.com/ShijieZhou-UCLA/feature-3dgs/issues/14#issuecomment-2185639700 Hope it can help you.

jyh-fort commented 4 months ago

same issue

WangPu1999 commented 1 day ago

Try this: #14 (comment) Hope it can help you.

Solved! Thank you!