Improbable-AI / walk-these-ways

Sim-to-real RL training and deployment tools for the Unitree Go1 robot.
https://gmargo11.github.io/walk-these-ways/
Other
492 stars 129 forks source link

TorchScript bug while testing #26

Closed ZJEast closed 1 year ago

ZJEast commented 1 year ago
/walk-these-ways-master$ python scripts/test.py
Importing module 'gym_38' (/home/user/code/walk-these-ways-master/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/user/code/walk-these-ways-master/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.10.0+cu113
Device count 1
/home/user/code/walk-these-ways-master/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/user/.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Emitting ninja build file /home/user/.cache/torch_extensions/py38_cu113/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
[Warning] [carb.gym.plugin] useGpuPipeline is set, forcing GPU PhysX
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0

Traceback (most recent call last):
  File "scripts/test.py", line 206, in <module>
    run_env(render=True, headless=False)
  File "scripts/test.py", line 188, in run_env
    env = VelocityTrackingEasyEnv(sim_device='cuda:0', headless=False, cfg=Cfg)
  File "/home/user/code/walk-these-ways-master/go1_gym/envs/go1/velocity_tracking/__init__.py", line 19, in __init__
    super().__init__(cfg, sim_params, physics_engine, sim_device, headless, eval_cfg, initial_dynamics_dict)
  File "/home/user/code/walk-these-ways-master/go1_gym/envs/base/legged_robot.py", line 44, in __init__
    super().__init__(self.cfg, sim_params, physics_engine, sim_device, headless, self.eval_cfg)
  File "/home/user/code/walk-these-ways-master/go1_gym/envs/base/base_task.py", line 71, in __init__
    self.create_sim()
  File "/home/user/code/walk-these-ways-master/go1_gym/envs/base/legged_robot.py", line 515, in create_sim
    self._create_envs()
  File "/home/user/code/walk-these-ways-master/go1_gym/envs/base/legged_robot.py", line 1555, in _create_envs
    pos[0:1] += torch_rand_float(-self.cfg.terrain.x_init_range, self.cfg.terrain.x_init_range, (1, 1),
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)
gmargo11 commented 1 year ago

Hi @ZJEast ,

I haven't seen this error before, it looks like the problem begins with the warning output MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0. You could try running that command as suggested (sysctl dev.i915.perf_stream_paranoid=0)

What hardware are you using to run this code? It looks like it could be a general issue with your TorchScript or isaacgym installation. Could you check whether the examples provided with the isaacgym simulator are working? (By running the python scripts in isaacgym/python/examples)

-Gabe

ZJEast commented 1 year ago

Thank you for your response so quickly. I think I do have some problems about isaacgym indeed.

python joint_monkey.py

Importing module 'gym_38' (/home/user/code/walk-these-ways-master/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/user/code/walk-these-ways-master/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
WARNING: Forcing CPU pipeline.
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: disabled
Loading asset 'mjcf/nv_humanoid.xml' from '../../assets'
DOF 0
  Name:     'abdomen_z'
  Type:     Rotation
  Stiffness:  20.0
  Damping:  5.0
  Armature:  0.02
  Limited?  True
    Lower   -0.785398
    Upper   0.785398
....

and nvidia-smi ‘s output

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090         Off| 00000000:01:00.0  On |                  Off |
|  0%   45C    P8               31W / 450W|    493MiB / 24564MiB |      6%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

i don't know whether this issue cause the torchscript error

ZJEast commented 1 year ago

Hi, I have already solve the problem about isaacgym, but the torchscript error still exists.

ZJEast commented 1 year ago

At the beginning, I follow README and install the 1.10.0 version of pytorch. Now I try to upgrade my pytorch version, and this problem has been solved.

tula-liuxin commented 6 months ago

At the beginning, I follow README and install the 1.10.0 version of pytorch. Now I try to upgrade my pytorch version, and this problem has been solved.

Hi, can I ask what specific commands you used to solve this? I also tried to upgrade pytorch version to both the latest version and 1.10.0 by: pip uninstall torch

To torch=1.10.0

pip install torch==1.10.0 //Or to latest torch pip install torch And both failed.