balakumar-s
commented
9 months ago Do you have NVIDIA driver 535 or above?
Closed D-1shu closed 3 months ago
balakumar-s
commented
9 months ago Do you have NVIDIA driver 535 or above?
D-1shu
commented
9 months ago @balakumar-s Driver Version: 525.60.11 CUDA Version: 12.0
balakumar-s
commented
9 months ago Can you try with Driver 535? We haven't tested on 525.
Car38639
commented
8 months ago I hit the same issue. I tried both 525 and 535 and neither of them worked.
anubhav-dogra
commented
7 months ago Same here, Error is same. My Nvidia Driver Version: 535.161.07 CUDA Version: 12.2
shengyangzhuang
commented
5 months ago I am encountering similar errors while trying to build a Docker image for Isaac Sim 2022.2.1. Any insights or suggestions to resolve this issue would be greatly appreciated!
szhuang@Y9000X:~$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)
szhuang@Y9000X:~$ nvidia-smi
Fri Jun 21 15:52:14 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06 Driver Version: 545.29.06 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 On | Off |
| 0% 44C P8 10W / 450W | 320MiB / 24564MiB | 3% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2291 G /usr/lib/xorg/Xorg 113MiB |
| 0 N/A N/A 2551 G /usr/bin/gnome-shell 125MiB |
| 0 N/A N/A 4183 G /opt/teamviewer/tv_bin/TeamViewer 47MiB |
+---------------------------------------------------------------------------------------+szhuang@Y9000X:~/curobo/docker$ bash build_docker.sh isaac_sim_2022.2.1
Building Isaac Sim docker
isaac_sim.dockerfile
[+] Building 3.5s (30/32) docker:default
=> [internal] load build definition from isaac_sim.dockerfile 0.0s
=> => transferring dockerfile: 8.77kB 0.0s
=> [internal] load metadata for nvcr.io/nvidia/cudagl:11.4.2-devel-ubuntu20.04 0.8s
=> [internal] load metadata for nvcr.io/nvidia/isaac-sim:2022.2.1 0.7s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 60B 0.0s
=> [isaac-sim 1/1] FROM nvcr.io/nvidia/isaac-sim:2022.2.1@sha256:48ccef298a8a7be20a3325a8e9c76546b5bc18ee1480e9594a5977cd2f04a54a 0.0s
=> [stage-1 1/27] FROM nvcr.io/nvidia/cudagl:11.4.2-devel-ubuntu20.04@sha256:28bfecb17a9f295124e985e575a84ba851cff435c23b48536e14406126ae4748 0.0s
=> CACHED [stage-1 2/27] RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections 0.0s
=> CACHED [stage-1 3/27] RUN apt-get update && apt-get install -y tzdata software-properties-common && rm -rf /var/lib/apt/lists/* && ln -fs /usr/share/zoneinfo/America/Los_Angeles /etc/localtim 0.0s
=> CACHED [stage-1 4/27] RUN apt-get update && apt-get install -y --no-install-recommends libatomic1 libegl1 libglu1-mesa libgomp1 libsm6 libxi6 libxrandr2 libxt6 lib 0.0s
=> CACHED [stage-1 5/27] RUN wget -q --show-progress --progress=bar:force:noscroll https://sdk.lunarg.com/sdk/download/1.3.224.1/linux/vulkansdk-linux-x86_64-1.3.224.1.tar.gz -O /tmp/vulkans 0.0s
=> CACHED [stage-1 6/27] COPY --from=isaac-sim /isaac-sim /isaac-sim 0.0s
=> CACHED [stage-1 7/27] RUN mkdir -p /root/.nvidia-omniverse/config 0.0s
=> CACHED [stage-1 8/27] COPY --from=isaac-sim /root/.nvidia-omniverse/config /root/.nvidia-omniverse/config 0.0s
=> CACHED [stage-1 9/27] COPY --from=isaac-sim /etc/vulkan/icd.d/nvidia_icd.json /etc/vulkan/icd.d/nvidia_icd.json 0.0s
=> CACHED [stage-1 10/27] COPY --from=isaac-sim /etc/vulkan/icd.d/nvidia_icd.json /etc/vulkan/implicit_layer.d/nvidia_layers.json 0.0s
=> CACHED [stage-1 11/27] WORKDIR /isaac-sim 0.0s
=> CACHED [stage-1 12/27] RUN echo "alias omni_python='/isaac-sim/python.sh'" >> /.bashrc 0.0s
=> CACHED [stage-1 13/27] RUN /isaac-sim/python.sh -m pip install "robometrics[evaluator] @ git+https://github.com/fishbotics/robometrics.git" 0.0s
=> CACHED [stage-1 14/27] RUN mkdir /pkgs && cd /pkgs && git clone https://github.com/NVlabs/curobo.git 0.0s
=> CACHED [stage-1 15/27] RUN /isaac-sim/python.sh -m pip install ninja wheel tomli 0.0s
=> CACHED [stage-1 16/27] RUN cd /pkgs/curobo && /isaac-sim/python.sh -m pip install .[dev] --no-build-isolation 0.0s
=> CACHED [stage-1 17/27] WORKDIR /pkgs/curobo 0.0s
=> CACHED [stage-1 18/27] RUN apt-get update && apt-get install -y curl tcl && rm -rf /var/lib/apt/lists/* 0.0s
=> CACHED [stage-1 19/27] RUN cd /pkgs && wget https://cmake.org/files/v3.27/cmake-3.27.1.tar.gz && tar -xvzf cmake-3.27.1.tar.gz && apt update && apt install -y build-essential checkinstall zli 0.0s
=> CACHED [stage-1 20/27] RUN cd /pkgs && git clone https://github.com/sqlite/sqlite.git -b version-3.39.4 && cd /pkgs/sqlite && CFLAGS=-fPIC ./configure --prefix=/pkgs/sqlite/install/ && make && 0.0s
=> CACHED [stage-1 21/27] RUN cd /pkgs && git clone https://github.com/google/glog.git -b v0.6.0 && cd glog && mkdir build && cd build && cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DC 0.0s
=> CACHED [stage-1 22/27] RUN cd /pkgs && git clone https://github.com/gflags/gflags.git -b v2.2.2 && cd gflags && mkdir build && cd build && cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE=ON 0.0s
=> CACHED [stage-1 23/27] RUN cd /pkgs && git clone https://github.com/valtsblukis/nvblox.git && cd /pkgs/nvblox/nvblox && mkdir build && cd build && cmake .. -DBUILD_REDISTRIBUTABLE=ON -DC 0.0s
=> CACHED [stage-1 24/27] RUN cd /pkgs/glog && mkdir build_isaac && cd build_isaac && cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DWITH_GFLAGS=OFF -DWITH_GTEST=OFF -DBUILD_SHARED_LIBS=OFF 0.0s
=> ERROR [stage-1 25/27] RUN cd /pkgs && git clone https://github.com/nvlabs/nvblox_torch.git && cd /pkgs/nvblox_torch && sh install_isaac_sim.sh $(/isaac-sim/python.sh -c 'import torch.utils; pr 2.6s
------
> [stage-1 25/27] RUN cd /pkgs && git clone https://github.com/nvlabs/nvblox_torch.git && cd /pkgs/nvblox_torch && sh install_isaac_sim.sh $(/isaac-sim/python.sh -c 'import torch.utils; print(torch.utils.cmake_prefix_path)') && /isaac-sim/python.sh -m pip install -e .:
0.201 Cloning into 'nvblox_torch'...
1.534 -- The CXX compiler identification is GNU 9.4.0
1.919 -- The CUDA compiler identification is NVIDIA 11.4.152
1.931 -- Detecting CXX compiler ABI info
1.982 -- Detecting CXX compiler ABI info - done
1.990 -- Check for working CXX compiler: /usr/bin/c++ - skipped
1.990 -- Detecting CXX compile features
1.991 -- Detecting CXX compile features - done
1.993 -- Detecting CUDA compiler ABI info
2.360 -- Detecting CUDA compiler ABI info - done
2.380 -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
2.381 -- Detecting CUDA compile features
2.381 -- Detecting CUDA compile features - done
2.384 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
2.433 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
2.433 -- Looking for pthread_create in pthreads
2.469 -- Looking for pthread_create in pthreads - not found
2.469 -- Looking for pthread_create in pthread
2.517 -- Looking for pthread_create in pthread - found
2.518 -- Found Threads: TRUE
2.529 -- Found CUDA: /usr/local/cuda (found version "11.4")
2.529 -- Caffe2: CUDA detected: 11.4
2.529 -- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
2.529 -- Caffe2: CUDA toolkit directory: /usr/local/cuda
2.589 -- Caffe2: Header version is: 11.4
2.590 -- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)
2.591 CMake Warning at /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/public/cuda.cmake:120 (message):
2.591 Caffe2: Cannot find cuDNN library. Turning the option off
2.591 Call Stack (most recent call first):
2.591 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/Caffe2Config.cmake:92 (include)
2.591 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
2.591 CMakeLists.txt:32 (find_package)
2.591
2.591
2.592 CMake Warning at /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/public/cuda.cmake:214 (message):
2.592 Failed to compute shorthash for libnvrtc.so
2.592 Call Stack (most recent call first):
2.592 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/Caffe2Config.cmake:92 (include)
2.592 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
2.592 CMakeLists.txt:32 (find_package)
2.592
2.592
2.592 CMake Warning at /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/public/utils.cmake:385 (message):
2.592 In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
2.592 to cmake instead of implicitly setting it as an env variable. This will
2.592 become a FATAL_ERROR in future version of pytorch.
2.592 Call Stack (most recent call first):
2.592 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/public/cuda.cmake:437 (torch_cuda_get_nvcc_gencode_flag)
2.592 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/Caffe2Config.cmake:92 (include)
2.592 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
2.592 CMakeLists.txt:32 (find_package)
2.592
2.592
2.592 -- Added CUDA NVCC flags for: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_70,code=compute_70
2.593 CMake Error at /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Caffe2/Caffe2Config.cmake:100 (message):
2.593 Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
2.593 libraries. Please set the proper cuDNN prefixes and / or install cuDNN.
2.593 Call Stack (most recent call first):
2.593 /isaac-sim/extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
2.593 CMakeLists.txt:32 (find_package)
2.593
2.593
2.593 -- Configuring incomplete, errors occurred!
------
isaac_sim.dockerfile:247
--------------------
246 |
247 | >>> RUN cd /pkgs && git clone https://github.com/nvlabs/nvblox_torch.git && \
248 | >>> cd /pkgs/nvblox_torch && \
249 | >>> sh install_isaac_sim.sh $($omni_python -c 'import torch.utils; print(torch.utils.cmake_prefix_path)') && \
250 | >>> $omni_python -m pip install -e .
251 |
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /pkgs && git clone https://github.com/nvlabs/nvblox_torch.git && cd /pkgs/nvblox_torch && sh install_isaac_sim.sh $($omni_python -c 'import torch.utils; print(torch.utils.cmake_prefix_path)') && $omni_python -m pip install -e ." did not complete successfully: exit code: 1szhuang@Y9000X:~/curobo/docker$ bash build_dev_docker.sh isaac_sim_2022.2.1
isaac_sim_2022.2.1
1000
[+] Building 0.7s (2/2) FINISHED docker:default
=> [internal] load build definition from user_isaac_sim.dockerfile 0.0s
=> => transferring dockerfile: 2.84kB 0.0s
=> ERROR [internal] load metadata for docker.io/library/curobo_docker:isaac_sim_2022.2.1 0.6s
------
> [internal] load metadata for docker.io/library/curobo_docker:isaac_sim_2022.2.1:
------
user_isaac_sim.dockerfile:14
--------------------
12 | # Check architecture and load:
13 | ARG IMAGE_TAG
14 | >>> FROM curobo_docker:${IMAGE_TAG}
15 | # Set variables
16 | ARG USERNAME
--------------------
ERROR: failed to solve: curobo_docker:isaac_sim_2022.2.1: failed to resolve source metadata for docker.io/library/curobo_docker:isaac_sim_2022.2.1: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failedWe have updated cuRobo to support Isaac Sim 4.0 as 2022.2.1 is no longer available.
Issue Details Error building in docker image from
bash build_docker.sh isaac_sim_2022.2.1command atstep 41