search

leggedrobotics / legged_gym

Isaac Gym Environments for Legged Robots
Other
1.22k stars 348 forks source link

Terrain and A1/Cassie only working on CPU #11

Closed sujitvasanth closed 2 years ago

sujitvasanth commented 2 years ago

OS Version: Ubuntu 21.04 Nvidia Driver: 495 Graphics: GTX 1660 Ti Pytorch: PyTorch version 1.10.1+cu102

Hi tried anymal_c_flat and works fine on GTX 1660 Ti using nvidia-driver-495 When i try to run anymal_c_rough only works on CPU pipeline..otherwise terminal says killed. Cassie works on cpu pipline python3 train.py --task=cassie --num_envs=900 --sim_device=cpu will not let me run rl_device=cuda

how do I get it all runnning on GPU or is my GPU not advanced enough?

nikitardn commented 2 years ago

This is probably because your GPU runs out of memory. In the appendix of our paper we list the minimal requirements on memory for different number of envs

sujitvasanth commented 2 years ago

Hi I upgraded to at 12Gb RTX3060 GPU, upgraded to latest Pytorch version and still same problem The IsaacGym examples all wok perfectly, can play A1 on gpu but cant train fully in gpu (terminal comes back as "Killed"

PyTorch version: 1.12.0+cu116 CUDA used to build PyTorch: 11.6 OS: Ubuntu 20.04.4 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Libc version: glibc-2.31 Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] (64-bit runtime) Python platform: Linux-5.13.0-52-generic-x86_64-with-glibc2.29 Is CUDA available: True GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Nvidia driver version: 510.73.08 [pip3] numpy==1.19.5 [pip3] torch==1.12.0+cu116 [pip3] torchaudio==0.12.0+cu116 [pip3] torchvision==0.13.0+cu116

i.e. ~/Desktop/legged_gym-master/legged_gym/scripts$ python train.py --task=a1 --num_envs=8 returns "Killed" surely this does not need more than 12 Gb?

python train.py --task=a1 --num_envs=64 --sim_device=cpu --rl_device=cuda:0 works

Or should I just give up on native python? I have tried pycharm virtual envoronment which just crashes when the same is tried ~(again it loads isaac gymn examples). Should I resort to conda?

sujitvasanth commented 2 years ago

OK managed to fix it by upgrading the nvidia driver on ubuntu add drivers to 515.48.07 now all examples run on gpu pipeline to do this on ubuntu go to seetings --> additional drivers --> and select: Nvidia deriver metapackage from nvidia-driver-515 (propritary, tested)

Modified setup: GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Nvidia driver version: 515.48.07

the rest of my setup: OS: Ubuntu 20.04.4 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Libc version: glibc-2.31 Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] (64-bit runtime) Python platform: Linux-5.13.0-52-generic-x86_64-with-glibc2.29 [pip3] numpy==1.19.5 [pip3] torch==1.12.0+cu116 [pip3] torchaudio==0.12.0+cu116 [pip3] torchvision==0.13.0+cu116 CUDA used to build PyTorch: 11.6 ROCM used to build PyTorch: N/A

now all examples run on gpu pipeline includong terrain