search

princeton-vl / DROID-SLAM

BSD 3-Clause "New" or "Revised" License
1.79k stars 299 forks source link

Runtime error: CUDA error: invalid configuration argument #133

Open estaudere opened 2 months ago

estaudere commented 2 months ago

Great repo! While running inference on some custom data (as part of shape-of-motion), I get this error

Traceback (most recent call last):
  File "/home/ubuntu/home/shape-of-motion/preproc/recon_with_depth.py", line 265, in <module>
    traj_est = droid.terminate(image_stream(args.img_dir, args.calib, args.stride))
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/droid.py", line 81, in terminate
    self.backend(7)
  File "/home/ubuntu/miniconda3/envs/som/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/droid_backend.py", line 39, in __call__
    graph.update_lowmem(steps=steps)
  File "/home/ubuntu/miniconda3/envs/som/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/factor_graph.py", line 293, in update_lowmem
    self.video.ba(target, weight, damping, self.ii, self.jj, 1, t, 
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/depth_video.py", line 190, in ba
    droid_backends.ba(self.poses, self.disps, self.intrinsics[0], self.disps_sens,
RuntimeError: CUDA error: invalid configuration argument
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Would love any advice on solving this! I am running this on an Nvidia A10G with 24GB memory.

tb2-sy commented 2 months ago

Great repo! While running inference on some custom data (as part of shape-of-motion), I get this error

Traceback (most recent call last):
  File "/home/ubuntu/home/shape-of-motion/preproc/recon_with_depth.py", line 265, in <module>
    traj_est = droid.terminate(image_stream(args.img_dir, args.calib, args.stride))
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/droid.py", line 81, in terminate
    self.backend(7)
  File "/home/ubuntu/miniconda3/envs/som/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/droid_backend.py", line 39, in __call__
    graph.update_lowmem(steps=steps)
  File "/home/ubuntu/miniconda3/envs/som/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/factor_graph.py", line 293, in update_lowmem
    self.video.ba(target, weight, damping, self.ii, self.jj, 1, t, 
  File "/home/ubuntu/home/shape-of-motion/preproc/DROID-SLAM/droid_slam/depth_video.py", line 190, in ba
    droid_backends.ba(self.poses, self.disps, self.intrinsics[0], self.disps_sens,
RuntimeError: CUDA error: invalid configuration argument
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Would love any advice on solving this! I am running this on an Nvidia A10G with 24GB memory.

Hi, have you solved this problem?

ThiruRJST commented 2 months ago

Hey Buddy, This seems like CUDA version conflict. Some unwanted argument is being passed to the cuda kernel, I Think since this repo is 2-3years old. They might have built it with very old cuda-toolkits. Try downgrading your CUDA Toolkit.