"RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain." when running code following library install

[jstmn]

1. cuRobo installation mode (choose from [python, isaac sim, docker python, docker isaac sim]): python, library installation
2. python version: 3.8
3. Isaac Sim version (if using): n/a

---

Issue Details

Hi there,

I'm trying to run the examples but am getting the following error:

(venv) jstm@yggdrasil:[~/Libraries/curobo]: python examples/ik_example.py 
Traceback (most recent call last):
  File "examples/ik_example.py", line 232, in <module>
    demo_basic_ik()
  File "examples/ik_example.py", line 62, in demo_basic_ik
    result = ik_solver.solve_batch(goal)
  File "/home/jstm/Libraries/curobo/src/curobo/wrap/reacher/ik_solver.py", line 537, in solve_batch
    return self.solve_from_solve_state(
  File "/home/jstm/Libraries/curobo/src/curobo/wrap/reacher/ik_solver.py", line 675, in solve_from_solve_state
    result = self.solver.solve(goal_buffer, coord_position_seed)
  File "/home/jstm/Libraries/curobo/src/curobo/wrap/wrap_base.py", line 138, in solve
    act_seq = self.optimize(seed, shift_steps=0)
  File "/home/jstm/Libraries/curobo/src/curobo/wrap/wrap_base.py", line 70, in optimize
    act_seq = opt.optimize(act_seq, shift_steps)
  File "/home/jstm/Libraries/curobo/src/curobo/opt/opt_base.py", line 93, in optimize
    out = self._optimize(opt_tensor, shift_steps, n_iters)
  File "/home/jstm/Libraries/curobo/src/curobo/opt/newton/newton_base.py", line 141, in _optimize
    best_q, best_cost, q, grad_q = self._call_opt_iters_graph(q, grad_q)
  File "/home/jstm/Libraries/curobo/src/curobo/opt/newton/newton_base.py", line 515, in _call_opt_iters_graph
    return self._opt_iters(q, grad_q)
  File "/home/jstm/Libraries/curobo/src/curobo/opt/newton/newton_base.py", line 167, in _opt_iters
    cost_n, q, grad_q = self._opt_step(q.detach(), grad_q.detach())
  File "/home/jstm/Libraries/curobo/src/curobo/opt/newton/newton_base.py", line 176, in _opt_step
    q_n, cost_n, grad_q_n = self._approx_line_search(q, grad_q)
  File "/home/jstm/Libraries/curobo/src/curobo/opt/newton/newton_base.py", line 419, in _approx_line_search
    return self._wolfe_line_search(x, step_direction)
  File "/home/jstm/Libraries/curobo/src/curobo/opt/newton/newton_base.py", line 327, in _wolfe_line_search
    (best_x_n, best_c_n, best_grad_n) = wolfe_line_search(
  File "/home/jstm/Libraries/curobo/src/curobo/curobolib/ls.py", line 60, in wolfe_line_search
    r = line_search_cu.line_search(
RuntimeError: CUDA error: the provided PTX was compiled with an unsupported toolchain.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

here's my cuda driver info:

(venv) jstm@yggdrasil:[~/Libraries/curobo]: /usr/local/cuda/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Sep__8_19:17:24_PDT_2023
Cuda compilation tools, release 12.3, V12.3.52
Build cuda_12.3.r12.3/compiler.33281558_0

and i'm using pytorch version 2.1.2. I saw that CUDA 11.7 is specified for install use with Isaac sim, is it required for a just python installation?

Any ideas on how to fix this?

Thanks!

• jeremy

[balakumar-s]

What does nvidia-smi report? You need to have a nvidia driver that supports cuda 12.3.

[jstmn]

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.116.04   Driver Version: 525.116.04   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:65:00.0 Off |                  Off |
|  0%   42C    P8    25W / 450W |    654MiB / 24564MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

I have 525. Does this mean I need to install 545.23.06+ and cuda 12.3? (per this table)

[balakumar-s]

Yes. Or downgrade your torch installation to cuda that your current driver supports (12.0 or lower)

[balakumar-s]

Were you able to fix this issue? If not, reopen it with more details.

[jstmn]

Yes I was, it ran fine after changing to torch 2.0.1.

Thanks!