search

Improbable-AI / walk-these-ways

Sim-to-real RL training and deployment tools for the Unitree Go1 robot.
https://gmargo11.github.io/walk-these-ways/
Other
488 stars 129 forks source link

CUDA error #77

Open wuyukun-tong opened 2 months ago

wuyukun-tong commented 2 months ago

when i try to run python train.py some error occurred! ubuntu 20.04 cuda 11.8 cudnn 8.8.1 pytorch 1.10 I'm on another 18.04 computer with the same version of cuda, cudnn, and pytorch, and it runs without errors!

ML_LOGGER_USER is not set. This is required for online usage. ✓ created a new logging client Dashboard: http://app.dash.ml/gait-conditioned-agility/2024-04-17/train/135613.659771 Log_directory: /home/wyk/Downloads/walk-these-ways-go2/runs Warning: failed to preload CUDA lib Warning: failed to preload PhysX libs Importing module 'gym_38' (/home/wyk/Downloads/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so) Setting GYM_USD_PLUG_INFO_PATH to /home/wyk/Downloads/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json PyTorch version 1.10.0+cu113 Device count 1 /home/wyk/Downloads/isaacgym/python/isaacgym/_bindings/src/gymtorch Using /home/wyk/.cache/torch_extensions/py38_cu113 as PyTorch extensions root... Emitting ninja build file /home/wyk/.cache/torch_extensions/py38_cu113/gymtorch/build.ninja... Building extension module gymtorch... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module gymtorch... [Warning] [carb.gym.plugin] useGpuPipeline is set, forcing GPU PhysX Not connected to PVD /buildAgent/work/99bede84aa0a52c2/source/physx/src/gpu/PxPhysXGpuModuleLoader.cpp (148) : internal error : libcuda.so!

[Warning] [carb.gym.plugin] Failed to create a PhysX CUDA Context Manager. Falling back to CPU. Physics Engine: PhysX Physics Device: cpu GPU Pipeline: disabled [Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137 [Error] [carb.gym.plugin] Must enable GPU pipeline to use state tensors [Error] [carb.gym.plugin] Must enable GPU pipeline to use state tensors Traceback (most recent call last): File "scripts/train.py", line 257, in train_go2(headless=False) File "scripts/train.py", line 208, in train_go2 env = VelocityTrackingEasyEnv(sim_device='cuda:0', headless=False, cfg=Cfg) File "/home/wyk/Downloads/walk-these-ways-go2/go2_gym/envs/go2/velocity_tracking/init.py", line 19, in init super().init(cfg, sim_params, physics_engine, sim_device, headless, eval_cfg, initial_dynamics_dict) File "/home/wyk/Downloads/walk-these-ways-go2/go2_gym/envs/base/legged_robot.py", line 51, in init self._init_buffers() File "/home/wyk/Downloads/walk-these-ways-go2/go2_gym/envs/base/legged_robot.py", line 1167, in _init_buffers self.noise_scale_vec = self._get_noise_scale_vec(self.cfg) # , self.eval_cfg) File "/home/wyk/Downloads/walk-these-ways-go2/go2_gym/envs/base/legged_robot.py", line 1118, in _get_noise_scale_vec noise_vec = noise_vec.to(self.device) RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.