ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory

[xbotaria]

Hi,

I am trying to inference the phi-3 model using this code, however I could not succeed because of importing this library.

When I import this lib, I am getting following error message:

from onnxruntime_genai.onnxruntime_genai import *
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory

I have installed all cuda related things and pytorch cuda is working as normal.

[natke]

Hi @xbotaria,

Can you please share your environment via pip list or similar?

Natalie

[baijumeswani]

from onnxruntime_genai.onnxruntime_genai import * ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory

Have you installed cuDNN? You can download and install cuDNN for cuda 11.x from here: https://developer.nvidia.com/rdp/cudnn-archive

I have installed all cuda related things and pytorch cuda is working as normal.

PyTorch binaries ship with a copy of the CUDA, cuDNN and related libs. Ort and Ort-GenAI depend on users to install the necessary CUDA, cuDNN toolkit/files.

[xbotaria]

@natke , thanks for your reply. Here is the list of installed packages

Click here to see installed packages

Package Version ------------------------- -------------------------- absl-py 2.1.0 aiohttp 3.9.3 aiosignal 1.3.1 annotated-types 0.6.0 apex 0.1 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 asttokens 2.4.1 astunparse 1.6.3 async-timeout 4.0.3 attrs 23.2.0 audioread 3.0.1 beautifulsoup4 4.12.3 bleach 6.1.0 blis 0.7.11 cachetools 5.3.3 catalogue 2.0.10 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 3.3.2 click 8.1.7 cloudpathlib 0.16.0 cloudpickle 3.0.0 cmake 3.28.3 comm 0.2.1 confection 0.1.4 contourpy 1.2.0 cuda-python 12.4.0rc7+3.ge75c8a9.dirty cudf 24.2.0 cudnn 1.1.2 cugraph 24.2.0 cugraph-dgl 24.2.0 cugraph-service-client 24.2.0 cugraph-service-server 24.2.0 cuml 24.2.0 cupy-cuda12x 13.0.0 cycler 0.12.1 cymem 2.0.8 Cython 3.0.9 dask 2024.1.1 dask-cuda 24.2.0 dask-cudf 24.2.0 debugpy 1.8.1 decorator 5.1.1 defusedxml 0.7.1 distributed 2024.1.1 dm-tree 0.1.8 einops 0.7.0 exceptiongroup 1.2.0 execnet 2.0.2 executing 2.0.1 expecttest 0.1.3 fastjsonschema 2.19.1 fastrlock 0.8.2 filelock 3.13.1 flash-attn 2.4.2 fonttools 4.49.0 frozenlist 1.4.1 fsspec 2024.2.0 gast 0.5.4 google-auth 2.28.2 google-auth-oauthlib 0.4.6 graphsurgeon 0.4.6 grpcio 1.62.1 hypothesis 5.35.1 idna 3.6 igraph 0.11.4 importlib_metadata 7.0.2 iniconfig 2.0.0 intel-openmp 2021.4.0 ipykernel 6.29.3 ipython 8.21.0 ipython-genutils 0.2.0 jedi 0.19.1 Jinja2 3.1.3 joblib 1.3.2 json5 0.9.22 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter_client 8.6.0 jupyter_core 5.7.1 jupyter-tensorboard 0.2.0 jupyterlab 2.3.2 jupyterlab_pygments 0.3.0 jupyterlab-server 1.2.0 jupytext 1.16.1 kiwisolver 1.4.5 langcodes 3.3.0 lark 1.1.9 lazy_loader 0.3 librosa 0.10.1 lightning-thunder 0.0.0.dev0 lightning-utilities 0.10.1 llvmlite 0.42.0 locket 1.0.0 looseversion 1.3.0 Markdown 3.5.2 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.8.3 matplotlib-inline 0.1.6 mdit-py-plugins 0.4.0 mdurl 0.1.2 mistune 3.0.2 mkl 2021.1.1 mkl-devel 2021.1.1 mkl-include 2021.1.1 mock 5.1.0 mpmath 1.3.0 msgpack 1.0.8 multidict 6.0.5 murmurhash 1.0.10 nbclient 0.9.0 nbconvert 7.16.2 nbformat 5.9.2 nest-asyncio 1.6.0 networkx 2.6.3 ninja 1.11.1.1 notebook 6.4.10 numba 0.59.0+1.g20ae2b56c numpy 1.24.4 nvfuser 0.1.5a0+f73ff1bc6a nvidia-dali-cuda120 1.35.0 nvidia-pyindex 1.0.9 nvtx 0.2.5 oauthlib 3.2.2 onnx 1.15.0rc2 onnxruntime-genai-cuda 0.2.0rc4 opencv 4.7.0 opt-einsum 3.3.0 optree 0.10.0 packaging 23.2 pandas 1.5.3 pandocfilters 1.5.1 parso 0.8.3 partd 1.4.1 pexpect 4.9.0 pillow 10.2.0 pip 24.0 platformdirs 4.2.0 pluggy 1.4.0 ply 3.11 polygraphy 0.49.7 pooch 1.8.1 preshed 3.0.9 prettytable 3.10.0 prometheus_client 0.20.0 prompt-toolkit 3.0.43 protobuf 4.24.4 psutil 5.9.4 ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 14.0.1 pyasn1 0.5.1 pyasn1-modules 0.3.0 pybind11 2.11.1 pybind11-global 2.11.1 pycocotools 2.0+nv0.8.0 pycparser 2.21 pydantic 2.6.3 pydantic_core 2.16.3 Pygments 2.17.2 pylibcugraph 24.2.0 pylibcugraphops 24.2.0 pylibraft 24.2.0 pynvjitlink 0.1.13 pynvml 11.4.1 pyparsing 3.1.2 pytest 8.0.2 pytest-flakefinder 1.1.0 pytest-rerunfailures 13.0 pytest-shard 0.1.2 pytest-xdist 3.5.0 python-dateutil 2.9.0.post0 python-hostlist 1.23.0 pytorch-quantization 2.1.2 pytorch-triton 2.2.0+e28a256d7 pytz 2024.1 PyYAML 6.0.1 pyzmq 25.1.2 raft-dask 24.2.0 rapids-dask-dependency 24.2.0a0 referencing 0.33.0 regex 2023.12.25 requests 2.31.0 requests-oauthlib 1.3.1 rich 13.7.1 rmm 24.2.0 rpds-py 0.18.0 rsa 4.9 scikit-learn 1.2.0 scipy 1.12.0 Send2Trash 1.8.2 setuptools 68.2.2 six 1.16.0 smart-open 6.4.0 sortedcontainers 2.4.0 soundfile 0.12.1 soupsieve 2.5 soxr 0.3.7 spacy 3.7.4 spacy-legacy 3.0.12 spacy-loggers 1.0.5 sphinx_glpi_theme 0.6 srsly 2.4.8 stack-data 0.6.3 sympy 1.12 tabulate 0.9.0 tbb 2021.11.0 tblib 3.0.0 tensorboard 2.9.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorrt 8.6.3 terminado 0.18.0 texttable 1.7.0 thinc 8.2.3 threadpoolctl 3.3.0 thriftpy2 0.4.17 tinycss2 1.2.1 toml 0.10.2 tomli 2.0.1 toolz 0.12.1 torch 2.3.0a0+40ec155e58.nv24.3 torch-tensorrt 2.3.0a0 torchdata 0.7.1a0 torchtext 0.17.0a0 torchvision 0.18.0a0 tornado 6.4 tqdm 4.66.2 traitlets 5.9.0 transformer-engine 1.4.0+0fbc76a treelite 4.0.0 typer 0.9.0 types-dataclasses 0.6.6 typing_extensions 4.10.0 ucx-py 0.36.0 uff 0.6.9 urllib3 1.26.18 wasabi 1.1.2 wcwidth 0.2.13 weasel 0.3.4 webencodings 0.5.1 Werkzeug 3.0.1 wheel 0.42.0 xdoctest 1.0.2 xgboost 2.0.3 yarl 1.9.4 zict 3.0.0 zipp 3.17.0

This is my development env, so there are many packages already installed. I am using a docker image of pytorch from nvcr.io. CUDA is perfectly working.

[natke]

Please see @baijumeswani''s message above

[Ben-Epstein]

Why does the library require version 8? I assume the issue (mine too) is that I have drives for cuda 9.1.

[baijumeswani]

ort-genai depends on ort lib. The cuda/cudnn requirements for ort are specified here: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html.

The variant of ort that is being used inside ort-genai is the cuda 11.8 variant. So, we require cuda 11.8 toolkit and cudnn 8.x for cuda 11.

[natke]

Hi @xbotaria, did you resolve your issue?

[xbotaria]

@natke , I could not find solution for this issue. Then I moved to huggingface inference code.

[guidevops]

I had this issue. i'm using debian, apt install cuda-toolkit fixed it for me.