Ubuntu 20.04 with cuda 11.8 kernal failing to install and crashing after the installation is done.

Your Talk at 2024 Nvidia GTC was great @balakumar-s . I am trying to get the curobo setup on my system with isaac sim to play around and potentially do some testing but I am having some issues with performance. Any input on below will be super helpfull.

The installation failed quite few times and I manged get it installed now but its crashing sometimes for curobo even after following instructions provided by documentation. Also is Isaac sim supposed to take liek 20 min just to warm and even after that its crashing ?

Is it possible to get a exact system specs and also the kernal specs on wihch the curobo has been tested ? It will be super helpful.

Here is my system specs:

cuRobo installation mode- isaac sim
python version: 3.7
Isaac Sim version : 2022.2.1
cuda - 11.8
nvidia-smi - 535.161.07
PyTorch - 2.2.1+cu121
GPU - NVIDIA GeForce RTX 3080
RAM - 32 GB

(base) mayur@mayur:~$ nvidia-smi
Tue Mar 26 16:32:57 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3080 ...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   50C    P5              24W /  35W |    388MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1898      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      2437      G   /usr/lib/xorg/Xorg                           28MiB |
|    0   N/A  N/A      4649    C+G   ...e/ov/pkg/isaac_sim-2022.2.1/kit/kit      332MiB |
+---------------------------------------------------------------------------------------+

(base) mayur@mayur:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

(base) mayur@mayur:~$ python
Python 3.11.7 (main, Dec 15 2023, 18:12:31) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
2.2.1+cu121

(base) mayur@mayur:~$ lscpu
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      39 bits physical, 48 bits virtual
CPU(s):                             16
On-line CPU(s) list:                0-15
Thread(s) per core:                 2
Core(s) per socket:                 8
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              141
Model name:                         11th Gen Intel(R) Core(TM) i9-11950H @ 2.60G
                                    Hz
Stepping:                           1
CPU MHz:                            2600.000
CPU max MHz:                        5000.0000
CPU min MHz:                        800.0000
BogoMIPS:                           5222.40
Virtualisation:                     VT-x
L1d cache:                          384 KiB
L1i cache:                          256 KiB
L2 cache:                           10 MiB
L3 cache:                           24 MiB
NUMA node0 CPU(s):                  0-15
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disable
                                    d via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __u
                                    ser pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional,
                                     RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep 
                                    mtrr pge mca cmov pat pse36 clflush dts acpi
                                     mmx fxsr sse sse2 ss ht tm pbe syscall nx p
                                    dpe1gb rdtscp lm constant_tsc art arch_perfm
                                    on pebs bts rep_good nopl xtopology nonstop_
                                    tsc cpuid aperfmperf tsc_known_freq pni pclm
                                    ulqdq dtes64 monitor ds_cpl vmx smx est tm2 
                                    ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 ss
                                    e4_2 x2apic movbe popcnt tsc_deadline_timer 
                                    aes xsave avx f16c rdrand lahf_lm abm 3dnowp
                                    refetch cpuid_fault epb cat_l2 invpcid_singl
                                    e cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced 
                                    tpr_shadow vnmi flexpriority ept vpid ept_ad
                                     fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erm
                                    s invpcid rdt_a avx512f avx512dq rdseed adx 
                                    smap avx512ifma clflushopt clwb intel_pt avx
                                    512cd sha_ni avx512bw avx512vl xsaveopt xsav
                                    ec xgetbv1 xsaves split_lock_detect dtherm i
                                    da arat pln pts hwp hwp_notify hwp_act_windo
                                    w hwp_epp hwp_pkg_req avx512vbmi umip pku os
                                    pke avx512_vbmi2 gfni vaes vpclmulqdq avx512
                                    _vnni avx512_bitalg tme avx512_vpopcntdq rdp
                                    id movdiri movdir64b fsrm avx512_vp2intersec
                                    t md_clear flush_l1d arch_capabilities

cpu isaac sim crashing

Oh sorry that was from different env. I am using 3.7 within conda base enviorment. (I attached the list below)

I could not find the environment.yml in curobo repo so I created my own custom coda env. Do you have the environment.yml for this repo ?

(base) mayur@mayur:~/Downloads$ pip list
Package                           Version
--------------------------------- ------------
aiobotocore                       2.7.0
aiohttp                           3.9.3
aioitertools                      0.7.1
aiosignal                         1.2.0
alabaster                         0.7.12
altair                            5.0.1
anaconda-anon-usage               0.4.3
anaconda-catalogs                 0.2.0
anaconda-client                   1.12.3
anaconda-cloud-auth               0.1.4
anaconda-navigator                2.5.2
anaconda-project                  0.11.1
anyio                             4.2.0
appdirs                           1.4.4
archspec                          0.2.3
argon2-cffi                       21.3.0
argon2-cffi-bindings              21.2.0
arrow                             1.2.3
astroid                           2.14.2
astropy                           5.3.4
asttokens                         2.0.5
async-lru                         2.0.4
atomicwrites                      1.4.0
attrs                             23.1.0
Automat                           20.2.0
autopep8                          1.6.0
Babel                             2.11.0
backports.functools-lru-cache     1.6.4
backports.tempfile                1.0
backports.weakref                 1.0.post1
bcrypt                            3.2.0
beautifulsoup4                    4.12.2
binaryornot                       0.4.4
black                             23.11.0
bleach                            4.1.0
blessed                           1.20.0
blinker                           1.6.2
bokeh                             3.3.4
boltons                           23.0.0
botocore                          1.31.64
Bottleneck                        1.3.7
Brotli                            1.0.9
cachetools                        4.2.2
certifi                           2024.2.2
cffi                              1.16.0
chardet                           4.0.0
charset-normalizer                2.0.4
click                             8.1.7
cloudpickle                       2.2.1
clyent                            1.2.2
colorama                          0.4.6
colorcet                          3.0.1
comm                              0.1.2
conda                             24.3.0
conda-build                       24.1.2
conda-content-trust               0.2.0
conda_index                       0.4.0
conda-libmamba-solver             24.1.0
conda-pack                        0.6.0
conda-package-handling            2.2.0
conda_package_streaming           0.9.0
conda-repo-cli                    1.0.75
conda-token                       0.4.0
conda-verify                      3.4.2
constantly                        23.10.4
contourpy                         1.2.0
cookiecutter                      2.5.0
cryptography                      42.0.2
cssselect                         1.2.0
cycler                            0.11.0
cytoolz                           0.12.2
dask                              2023.11.0
datashader                        0.16.0
debugpy                           1.6.7
decorator                         5.1.1
defusedxml                        0.7.1
diff-match-patch                  20200713
dill                              0.3.7
distributed                       2023.11.0
distro                            1.8.0
docstring-to-markdown             0.11
docutils                          0.18.1
entrypoints                       0.4
et-xmlfile                        1.1.0
executing                         0.8.3
fastjsonschema                    2.16.2
filelock                          3.13.1
flake8                            6.0.0
Flask                             2.2.5
fonttools                         4.25.0
frozenlist                        1.4.0
fsspec                            2023.10.0
future                            0.18.3
gensim                            4.3.0
gitdb                             4.0.7
GitPython                         3.1.37
gmpy2                             2.1.2
gpustat                           1.1.1
greenlet                          3.0.1
h5py                              3.9.0
HeapDict                          1.0.1
holoviews                         1.18.3
hvplot                            0.9.2
hyperlink                         21.0.0
idna                              3.4
imagecodecs                       2023.1.23
imageio                           2.33.1
imagesize                         1.4.1
imbalanced-learn                  0.11.0
importlib-metadata                7.0.1
incremental                       22.10.0
inflection                        0.5.1
iniconfig                         1.1.1
intake                            0.6.8
intervaltree                      3.1.0
ipykernel                         6.28.0
ipython                           8.20.0
ipython-genutils                  0.2.0
ipywidgets                        7.6.5
isort                             5.9.3
itemadapter                       0.3.0
itemloaders                       1.1.0
itsdangerous                      2.0.1
jaraco.classes                    3.2.1
jedi                              0.18.1
jeepney                           0.7.1
jellyfish                         1.0.1
Jinja2                            3.1.3
jmespath                          1.0.1
joblib                            1.2.0
json5                             0.9.6
jsonpatch                         1.32
jsonpointer                       2.1
jsonschema                        4.19.2
jsonschema-specifications         2023.7.1
jupyter                           1.0.0
jupyter_client                    8.6.0
jupyter-console                   6.6.3
jupyter_core                      5.5.0
jupyter-events                    0.8.0
jupyter-lsp                       2.2.0
jupyter_server                    2.10.0
jupyter_server_terminals          0.4.4
jupyterlab                        4.0.11
jupyterlab-pygments               0.1.2
jupyterlab_server                 2.25.1
jupyterlab-widgets                3.0.9
keyring                           23.13.1
kiwisolver                        1.4.4
lazy_loader                       0.3
lazy-object-proxy                 1.6.0
lckr_jupyterlab_variableinspector 3.1.0
libarchive-c                      2.9
libmambapy                        1.5.6
linkify-it-py                     2.0.0
llvmlite                          0.42.0
lmdb                              1.4.1
locket                            1.0.0
lxml                              4.9.3
lz4                               4.3.2
Markdown                          3.4.1
markdown-it-py                    2.2.0
MarkupSafe                        2.1.3
matplotlib                        3.8.0
matplotlib-inline                 0.1.6
mccabe                            0.7.0
mdit-py-plugins                   0.3.0
mdurl                             0.1.0
menuinst                          2.0.2
mistune                           2.0.4
mkl-fft                           1.3.8
mkl-random                        1.2.4
mkl-service                       2.4.0
more-itertools                    10.1.0
mpmath                            1.3.0
msgpack                           1.0.3
multidict                         6.0.4
multipledispatch                  0.6.0
munkres                           1.1.4
mypy                              1.8.0
mypy-extensions                   1.0.0
navigator-updater                 0.4.0
nbclient                          0.8.0
nbconvert                         7.10.0
nbformat                          5.9.2
nest-asyncio                      1.6.0
networkx                          3.1
nltk                              3.8.1
notebook                          7.0.8
notebook_shim                     0.2.3
numba                             0.59.0
numexpr                           2.8.7
numpy                             1.26.4
numpydoc                          1.5.0
nvidia-cublas-cu12                12.1.3.1
nvidia-cuda-cupti-cu12            12.1.105
nvidia-cuda-nvrtc-cu12            12.1.105
nvidia-cuda-runtime-cu12          12.1.105
nvidia-cudnn-cu12                 8.9.2.26
nvidia-cufft-cu12                 11.0.2.54
nvidia-curand-cu12                10.3.2.106
nvidia-cusolver-cu12              11.4.5.107
nvidia-cusparse-cu12              12.1.0.106
nvidia-ml-py                      12.535.133
nvidia-nccl-cu12                  2.19.3
nvidia-nvjitlink-cu12             12.4.99
nvidia-nvtx-cu12                  12.1.105
openpyxl                          3.0.10
overrides                         7.4.0
packaging                         23.1
pandas                            2.1.4
pandocfilters                     1.5.0
panel                             1.3.8
param                             2.0.2
parsel                            1.8.1
parso                             0.8.3
partd                             1.4.1
pathlib                           1.0.1
pathspec                          0.10.3
patsy                             0.5.3
pexpect                           4.8.0
pickleshare                       0.7.5
pillow                            10.2.0
pip                               23.3.1
pkce                              1.0.3
pkginfo                           1.9.6
platformdirs                      3.10.0
plotly                            5.9.0
pluggy                            1.0.0
ply                               3.11
prometheus-client                 0.14.1
prompt-toolkit                    3.0.43
Protego                           0.1.16
protobuf                          3.20.3
psutil                            5.9.0
ptyprocess                        0.7.0
pure-eval                         0.2.2
py-cpuinfo                        9.0.0
pyarrow                           14.0.2
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pycodestyle                       2.10.0
pycosat                           0.6.6
pycparser                         2.21
pyct                              0.5.0
pycurl                            7.45.2
pydantic                          1.10.12
pydeck                            0.8.0
PyDispatcher                      2.0.5
pydocstyle                        6.3.0
pyerfa                            2.0.0
pyflakes                          3.0.1
Pygments                          2.15.1
PyJWT                             2.4.0
pylint                            2.16.2
pylint-venv                       2.3.0
pyls-spyder                       0.4.0
pyodbc                            5.0.1
pyOpenSSL                         24.0.0
pyparsing                         3.0.9
PyQt5                             5.15.10
PyQt5-sip                         12.13.0
PyQtWebEngine                     5.15.6
PySocks                           1.7.1
pytest                            7.4.0
python-dateutil                   2.8.2
python-dotenv                     0.21.0
python-json-logger                2.0.7
python-lsp-black                  1.2.1
python-lsp-jsonrpc                1.0.0
python-lsp-server                 1.7.2
python-slugify                    5.0.2
python-snappy                     0.6.1
pytoolconfig                      1.2.6
pytz                              2023.3.post1
pyviz_comms                       3.0.0
pywavelets                        1.5.0
pyxdg                             0.27
PyYAML                            6.0.1
pyzmq                             25.1.2
QDarkStyle                        3.0.2
qstylizer                         0.2.2
QtAwesome                         1.2.2
qtconsole                         5.4.2
QtPy                              2.4.1
queuelib                          1.6.2
referencing                       0.30.2
regex                             2023.10.3
requests                          2.31.0
requests-file                     1.5.1
requests-toolbelt                 1.0.0
rfc3339-validator                 0.1.4
rfc3986-validator                 0.1.1
rich                              13.3.5
rope                              1.7.0
rpds-py                           0.10.6
Rtree                             1.0.1
ruamel.yaml                       0.17.21
ruamel-yaml-conda                 0.17.21
s3fs                              2023.10.0
scikit-image                      0.22.0
scikit-learn                      1.2.2
scipy                             1.11.4
Scrapy                            2.8.0
seaborn                           0.12.2
SecretStorage                     3.3.1
semver                            2.13.0
Send2Trash                        1.8.2
service-identity                  18.1.0
setuptools                        68.2.2
sip                               6.7.12
six                               1.16.0
smart-open                        5.2.1
smmap                             4.0.0
sniffio                           1.3.0
snowballstemmer                   2.2.0
sortedcontainers                  2.4.0
soupsieve                         2.5
Sphinx                            5.0.2
sphinxcontrib-applehelp           1.0.2
sphinxcontrib-devhelp             1.0.2
sphinxcontrib-htmlhelp            2.0.0
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-qthelp              1.0.3
sphinxcontrib-serializinghtml     1.1.5
spyder                            5.4.3
spyder-kernels                    2.4.4
SQLAlchemy                        2.0.25
stack-data                        0.2.0
statsmodels                       0.14.0
streamlit                         1.30.0
sympy                             1.12
tables                            3.9.2
tabulate                          0.9.0
tblib                             1.7.0
tenacity                          8.2.2
terminado                         0.17.1
terminator                        0.0.0
text-unidecode                    1.3
textdistance                      4.2.1
threadpoolctl                     2.2.0
three-merge                       0.1.1
tifffile                          2023.4.12
tinycss2                          1.2.1
tldextract                        3.2.0
toml                              0.10.2
tomlkit                           0.11.1
toolz                             0.12.0
torch                             2.2.1
tornado                           6.3.3
tqdm                              4.65.0
traitlets                         5.7.1
triton                            2.2.0
truststore                        0.8.0
Twisted                           23.10.0
typing_extensions                 4.9.0
tzdata                            2023.3
tzlocal                           2.1
uc-micro-py                       1.0.1
ujson                             5.4.0
Unidecode                         1.2.0
urllib3                           2.0.7
validators                        0.18.2
w3lib                             2.1.2
watchdog                          2.1.6
wcwidth                           0.2.5
webencodings                      0.5.1
websocket-client                  0.58.0
Werkzeug                          2.2.3
whatthepatch                      1.0.2
wheel                             0.41.2
widgetsnbextension                3.5.2
wrapt                             1.14.1
wurlitzer                         3.0.2
xarray                            2023.6.0
xyzservices                       2022.9.0
yapf                              0.31.0
yarl                              1.9.3
zict                              3.0.0
zipp                              3.17.0
zope.interface                    5.4.0
zstandard                         0.19.0

isaac sim

NVlabs / curobo

Ubuntu 20.04 with cuda 11.8 kernal failing to install and crashing after the installation is done. #201