[BUG/Help] OSError nvidia/cublas/lib/libcublas.so.11: symbol cublasLtGetStatusString, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

[jacks808]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

python cli_demo.py

Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Traceback (most recent call last):
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 172, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: symbol cublasLtGetStatusString, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cli_demo.py", line 8, in <module>
    tokenizer = AutoTokenizer.from_pretrained("/home/aistudio/ChatGLM-6B/", trust_remote_code=True)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 635, in from_pretrained
    pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py", line 915, in from_pretrained
    return config_class.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/transformers/configuration_utils.py", line 553, in from_pretrained
    return cls.from_dict(config_dict, **kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/transformers/configuration_utils.py", line 696, in from_dict
    config = cls(**config_dict)
  File "/home/aistudio/.cache/huggingface/modules/transformers_modules/configuration_chatglm.py", line 102, in __init__
    **kwargs
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/transformers/configuration_utils.py", line 336, in __init__
    import torch
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 217, in <module>
    _load_global_deps()
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 178, in _load_global_deps
    _preload_cuda_deps()
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 158, in _preload_cuda_deps
    ctypes.CDLL(cublas_path)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/nvidia/cublas/lib/libcublas.so.11: symbol cublasLtGetStatusString, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

Expected Behavior

No response

Steps To Reproduce

python cli_demo.py

Environment

- OS: linux for baidu aistudio
- Python: Python 3.7.4
- Transformers:  4.27.1
- PyTorch: 1.13.1
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) : return error:

Traceback (most recent call last):
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 172, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: symbol cublasLtGetStatusString, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 217, in <module>
    _load_global_deps()
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 178, in _load_global_deps
    _preload_cuda_deps()
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/torch/__init__.py", line 158, in _preload_cuda_deps
    ctypes.CDLL(cublas_path)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/nvidia/cublas/lib/libcublas.so.11: symbol cublasLtGetStatusString, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

more infos, pip installed packages are:

Package                        Version
------------------------------ ---------------
absl-py                        0.8.1
aiofiles                       23.1.0
aiohttp                        3.8.3
aiosignal                      1.2.0
alembic                        1.8.1
altair                         4.2.0
anyio                          3.6.1
argon2-cffi                    21.3.0
argon2-cffi-bindings           21.2.0
aspy.yaml                      1.3.0
astor                          0.8.1
astroid                        2.4.1
async-generator                1.10
async-timeout                  4.0.2
asynctest                      0.13.0
attrs                          19.2.0
audioread                      2.1.8
autopep8                       1.6.0
Babel                          2.8.0
backcall                       0.1.0
backports.zoneinfo             0.2.1
bce-python-sdk                 0.8.53
beautifulsoup4                 4.11.1
bleach                         3.1.0
blinker                        1.5
cachetools                     4.0.0
certifi                        2019.9.11
certipy                        0.1.3
cffi                           1.14.0
cfgv                           2.0.1
chardet                        3.0.4
charset-normalizer             2.1.1
click                          8.0.4
cloudpickle                    1.6.0
cma                            2.7.0
colorama                       0.4.4
colorlog                       4.1.0
commonmark                     0.9.1
cpm-kernels                    1.0.11
cryptography                   38.0.1
cycler                         0.10.0
Cython                         0.29
datasets                       2.7.0
debugpy                        1.6.0
decorator                      4.4.2
defusedxml                     0.7.1
dill                           0.3.3
easydict                       1.9
entrypoints                    0.3
et-xmlfile                     1.0.1
fastapi                        0.95.0
fastjsonschema                 2.16.1
ffmpy                          0.3.0
filelock                       3.0.12
fire                           0.5.0
flake8                         4.0.1
Flask                          1.1.1
Flask-Babel                    1.0.0
Flask-Cors                     3.0.8
forbiddenfruit                 0.1.3
frozenlist                     1.3.0
fsspec                         2022.11.0
funcsigs                       1.0.2
future                         0.18.0
gast                           0.3.3
gitdb                          4.0.5
GitPython                      3.1.14
google-auth                    1.10.0
google-auth-oauthlib           0.4.1
gradio                         3.19.1
graphviz                       0.13
greenlet                       1.1.3
grpcio                         1.35.0
gunicorn                       20.0.4
gym                            0.12.1
h11                            0.14.0
h5py                           2.9.0
httpcore                       0.16.3
httpx                          0.23.3
huggingface-hub                0.11.0
identify                       1.4.10
idna                           2.8
imageio                        2.6.1
imageio-ffmpeg                 0.3.0
importlib-metadata             0.23
importlib-resources            5.9.0
ipykernel                      6.9.1
ipython                        7.34.0
ipython-genutils               0.2.0
ipywidgets                     7.6.5
isort                          4.3.21
itsdangerous                   1.1.0
jdcal                          1.4.1
jedi                           0.17.2
jieba                          0.42.1
Jinja2                         3.0.0
joblib                         0.14.1
JPype1                         0.7.2
json5                          0.9.5
jsonschema                     3.1.1
jupyter-archive                3.2.1
jupyter-client                 5.3.3
jupyter-core                   4.11.1
jupyter-lsp                    1.5.1
jupyter-server                 1.16.0
jupyter-telemetry              0.1.0
jupyterhub                     1.3.0
jupyterlab                     3.4.5
jupyterlab-language-pack-zh-CN 3.4.post1
jupyterlab-pygments            0.2.2
jupyterlab-server              2.10.3
jupyterlab-widgets             3.0.3
kiwisolver                     1.1.0
latex2mathml                   3.75.2
lazy-object-proxy              1.4.3
librosa                        0.7.2
lightgbm                       3.1.1
linkify-it-py                  2.0.0
llvmlite                       0.31.0
lxml                           4.9.1
Mako                           1.2.2
Markdown                       3.1.1
markdown-it-py                 2.2.0
MarkupSafe                     2.0.1
matplotlib                     2.2.3
matplotlib-inline              0.1.6
mccabe                         0.6.1
mdit-py-plugins                0.3.3
mdtex2html                     1.2.0
mdurl                          0.1.1
mistune                        0.8.4
more-itertools                 7.2.0
moviepy                        1.0.1
multidict                      6.0.2
multiprocess                   0.70.11.1
nbclassic                      0.3.1
nbclient                       0.5.13
nbconvert                      5.3.1
nbformat                       4.4.0
nest-asyncio                   1.5.5
netifaces                      0.10.9
networkx                       2.4
nltk                           3.4.5
nodeenv                        1.3.4
notebook                       5.7.0
numba                          0.48.0
numpy                          1.19.5
nvidia-cublas-cu11             11.10.3.66
nvidia-cuda-nvrtc-cu11         11.7.99
nvidia-cuda-runtime-cu11       11.7.99
nvidia-cudnn-cu11              8.5.0.96
oauthlib                       3.1.0
objgraph                       3.4.1
opencv-python                  4.6.0.66
openpyxl                       3.0.5
opt-einsum                     3.3.0
orjson                         3.8.7
packaging                      20.9
paddle-bfloat                  0.1.7
paddle2onnx                    1.0.0
paddlefsl                      1.1.0
paddlehub                      2.3.0
paddlenlp                      2.4.2
paddlepaddle-gpu               2.4.0.post112
pamela                         1.0.0
pandas                         1.1.5
pandocfilters                  1.4.2
parl                           1.4.1
parso                          0.7.1
pathlib                        1.0.1
pexpect                        4.7.0
pickleshare                    0.7.5
Pillow                         8.2.0
pip                            23.1.2
pkgutil_resolve_name           1.3.10
plotly                         5.8.0
pluggy                         1.0.0
pre-commit                     1.21.0
prettytable                    0.7.2
proglog                        0.1.9
prometheus-client              0.14.1
prompt-toolkit                 2.0.10
protobuf                       3.20.0
psutil                         5.7.2
ptyprocess                     0.6.0
py4j                           0.10.9.2
pyarrow                        10.0.0
pyasn1                         0.4.8
pyasn1-modules                 0.2.7
pybboxes                       0.1.1
pycodestyle                    2.8.0
pycparser                      2.19
pycryptodome                   3.9.9
pydantic                       1.10.6
pydeck                         0.8.0
pydocstyle                     5.0.2
pydub                          0.25.1
pyflakes                       2.4.0
pyglet                         1.4.5
Pygments                       2.13.0
pyhumps                        3.8.0
pylint                         2.5.2
Pympler                        1.0.1
pynvml                         8.0.4
pyOpenSSL                      22.0.0
pyparsing                      2.4.2
pypmml                         0.9.11
pyrsistent                     0.15.4
python-dateutil                2.8.0
python-json-logger             2.0.4
python-jsonrpc-server          0.3.4
python-language-server         0.33.0
python-lsp-jsonrpc             1.0.0
python-lsp-server              1.5.0
python-multipart               0.0.6
pytz                           2019.3
pytz-deprecation-shim          0.1.0.post0
PyYAML                         5.1.2
pyzmq                          18.1.1
rarfile                        3.1
recordio                       0.1.7
regex                          2023.5.5
requests                       2.24.0
requests-oauthlib              1.3.0
resampy                        0.2.2
responses                      0.18.0
rfc3986                        1.5.0
rich                           12.6.0
rope                           0.17.0
rsa                            4.0
ruamel.yaml                    0.17.21
ruamel.yaml.clib               0.2.6
sahi                           0.10.1
scikit-learn                   0.24.2
scipy                          1.6.3
seaborn                        0.10.0
semver                         2.13.0
Send2Trash                     1.5.0
sentencepiece                  0.1.96
seqeval                        1.2.2
setuptools                     56.2.0
shapely                        2.0.0
shellcheck-py                  0.7.1.1
simplegeneric                  0.8.1
six                            1.15.0
sklearn                        0.0
smmap                          3.0.5
sniffio                        1.3.0
snowballstemmer                2.0.0
SoundFile                      0.10.3.post1
soupsieve                      2.3.2.post1
SQLAlchemy                     1.4.41
starlette                      0.26.1
streamlit                      1.13.0
streamlit-image-comparison     0.0.3
tabulate                       0.8.3
tb-nightly                     1.15.0a20190801
tb-paddle                      0.3.6
tenacity                       8.0.1
tensorboard                    2.1.0
tensorboardX                   1.8
termcolor                      1.1.0
terminado                      0.15.0
terminaltables                 3.1.10
testpath                       0.4.2
threadpoolctl                  2.1.0
tinycss2                       1.1.1
tokenizers                     0.13.3
toml                           0.10.0
toolz                          0.12.0
torch                          1.13.1
tornado                        5.1.1
tqdm                           4.64.1
traitlets                      5.4.0
transformers                   4.27.1
typed-ast                      1.4.1
typing_extensions              4.3.0
tzdata                         2022.7
tzlocal                        4.2
uc-micro-py                    1.0.1
ujson                          1.35
urllib3                        1.25.11
uvicorn                        0.21.1
validators                     0.20.0
virtualenv                     16.7.9
visualdl                       2.4.0
watchdog                       2.2.0
wcwidth                        0.1.7
webencodings                   0.5.1
websocket-client               1.4.1
websockets                     10.4
Werkzeug                       0.16.0
whatthepatch                   1.0.2
wheel                          0.36.2
widgetsnbextension             3.5.2
wrapt                          1.12.1
xarray                         0.16.2
xgboost                        1.3.3
xlrd                           1.2.0
xxhash                         3.1.0
yapf                           0.26.0
yarl                           1.7.2
zipp                           3.8.1

conda list:

Package                        Version
------------------------------ ---------------
absl-py                        0.8.1
aiofiles                       23.1.0
aiohttp                        3.8.3
aiosignal                      1.2.0
alembic                        1.8.1
altair                         4.2.0
anyio                          3.6.1
argon2-cffi                    21.3.0
argon2-cffi-bindings           21.2.0
aspy.yaml                      1.3.0
astor                          0.8.1
astroid                        2.4.1
async-generator                1.10
async-timeout                  4.0.2
asynctest                      0.13.0
attrs                          19.2.0
audioread                      2.1.8
autopep8                       1.6.0
Babel                          2.8.0
backcall                       0.1.0
backports.zoneinfo             0.2.1
bce-python-sdk                 0.8.53
beautifulsoup4                 4.11.1
bleach                         3.1.0
blinker                        1.5
cachetools                     4.0.0
certifi                        2019.9.11
certipy                        0.1.3
cffi                           1.14.0
cfgv                           2.0.1
chardet                        3.0.4
charset-normalizer             2.1.1
click                          8.0.4
cloudpickle                    1.6.0
cma                            2.7.0
colorama                       0.4.4
colorlog                       4.1.0
commonmark                     0.9.1
cpm-kernels                    1.0.11
cryptography                   38.0.1
cycler                         0.10.0
Cython                         0.29
datasets                       2.7.0
debugpy                        1.6.0
decorator                      4.4.2
defusedxml                     0.7.1
dill                           0.3.3
easydict                       1.9
entrypoints                    0.3
et-xmlfile                     1.0.1
fastapi                        0.95.0
fastjsonschema                 2.16.1
ffmpy                          0.3.0
filelock                       3.0.12
fire                           0.5.0
flake8                         4.0.1
Flask                          1.1.1
Flask-Babel                    1.0.0
Flask-Cors                     3.0.8
forbiddenfruit                 0.1.3
frozenlist                     1.3.0
fsspec                         2022.11.0
funcsigs                       1.0.2
future                         0.18.0
gast                           0.3.3
gitdb                          4.0.5
GitPython                      3.1.14
google-auth                    1.10.0
google-auth-oauthlib           0.4.1
gradio                         3.19.1
graphviz                       0.13
greenlet                       1.1.3
grpcio                         1.35.0
gunicorn                       20.0.4
gym                            0.12.1
h11                            0.14.0
h5py                           2.9.0
httpcore                       0.16.3
httpx                          0.23.3
huggingface-hub                0.11.0
identify                       1.4.10
idna                           2.8
imageio                        2.6.1
imageio-ffmpeg                 0.3.0
importlib-metadata             0.23
importlib-resources            5.9.0
ipykernel                      6.9.1
ipython                        7.34.0
ipython-genutils               0.2.0
ipywidgets                     7.6.5
isort                          4.3.21
itsdangerous                   1.1.0
jdcal                          1.4.1
jedi                           0.17.2
jieba                          0.42.1
Jinja2                         3.0.0
joblib                         0.14.1
JPype1                         0.7.2
json5                          0.9.5
jsonschema                     3.1.1
jupyter-archive                3.2.1
jupyter-client                 5.3.3
jupyter-core                   4.11.1
jupyter-lsp                    1.5.1
jupyter-server                 1.16.0
jupyter-telemetry              0.1.0
jupyterhub                     1.3.0
jupyterlab                     3.4.5
jupyterlab-language-pack-zh-CN 3.4.post1
jupyterlab-pygments            0.2.2
jupyterlab-server              2.10.3
jupyterlab-widgets             3.0.3
kiwisolver                     1.1.0
latex2mathml                   3.75.2
lazy-object-proxy              1.4.3
librosa                        0.7.2
lightgbm                       3.1.1
linkify-it-py                  2.0.0
llvmlite                       0.31.0
lxml                           4.9.1
Mako                           1.2.2
Markdown                       3.1.1
markdown-it-py                 2.2.0
MarkupSafe                     2.0.1
matplotlib                     2.2.3
matplotlib-inline              0.1.6
mccabe                         0.6.1
mdit-py-plugins                0.3.3
mdtex2html                     1.2.0
mdurl                          0.1.1
mistune                        0.8.4
more-itertools                 7.2.0
moviepy                        1.0.1
multidict                      6.0.2
multiprocess                   0.70.11.1
nbclassic                      0.3.1
nbclient                       0.5.13
nbconvert                      5.3.1
nbformat                       4.4.0
nest-asyncio                   1.5.5
netifaces                      0.10.9
networkx                       2.4
nltk                           3.4.5
nodeenv                        1.3.4
notebook                       5.7.0
numba                          0.48.0
numpy                          1.19.5
nvidia-cublas-cu11             11.10.3.66
nvidia-cuda-nvrtc-cu11         11.7.99
nvidia-cuda-runtime-cu11       11.7.99
nvidia-cudnn-cu11              8.5.0.96
oauthlib                       3.1.0
objgraph                       3.4.1
opencv-python                  4.6.0.66
openpyxl                       3.0.5
opt-einsum                     3.3.0
orjson                         3.8.7
packaging                      20.9
paddle-bfloat                  0.1.7
paddle2onnx                    1.0.0
paddlefsl                      1.1.0
paddlehub                      2.3.0
paddlenlp                      2.4.2
paddlepaddle-gpu               2.4.0.post112
pamela                         1.0.0
pandas                         1.1.5
pandocfilters                  1.4.2
parl                           1.4.1
parso                          0.7.1
pathlib                        1.0.1
pexpect                        4.7.0
pickleshare                    0.7.5
Pillow                         8.2.0
pip                            23.1.2
pkgutil_resolve_name           1.3.10
plotly                         5.8.0
pluggy                         1.0.0
pre-commit                     1.21.0
prettytable                    0.7.2
proglog                        0.1.9
prometheus-client              0.14.1
prompt-toolkit                 2.0.10
protobuf                       3.20.0
psutil                         5.7.2
ptyprocess                     0.6.0
py4j                           0.10.9.2
pyarrow                        10.0.0
pyasn1                         0.4.8
pyasn1-modules                 0.2.7
pybboxes                       0.1.1
pycodestyle                    2.8.0
pycparser                      2.19
pycryptodome                   3.9.9
pydantic                       1.10.6
pydeck                         0.8.0
pydocstyle                     5.0.2
pydub                          0.25.1
pyflakes                       2.4.0
pyglet                         1.4.5
Pygments                       2.13.0
pyhumps                        3.8.0
pylint                         2.5.2
Pympler                        1.0.1
pynvml                         8.0.4
pyOpenSSL                      22.0.0
pyparsing                      2.4.2
pypmml                         0.9.11
pyrsistent                     0.15.4
python-dateutil                2.8.0
python-json-logger             2.0.4
python-jsonrpc-server          0.3.4
python-language-server         0.33.0
python-lsp-jsonrpc             1.0.0
python-lsp-server              1.5.0
python-multipart               0.0.6
pytz                           2019.3
pytz-deprecation-shim          0.1.0.post0
PyYAML                         5.1.2
pyzmq                          18.1.1
rarfile                        3.1
recordio                       0.1.7
regex                          2023.5.5
requests                       2.24.0
requests-oauthlib              1.3.0
resampy                        0.2.2
responses                      0.18.0
rfc3986                        1.5.0
rich                           12.6.0
rope                           0.17.0
rsa                            4.0
ruamel.yaml                    0.17.21
ruamel.yaml.clib               0.2.6
sahi                           0.10.1
scikit-learn                   0.24.2
scipy                          1.6.3
seaborn                        0.10.0
semver                         2.13.0
Send2Trash                     1.5.0
sentencepiece                  0.1.96
seqeval                        1.2.2
setuptools                     56.2.0
shapely                        2.0.0
shellcheck-py                  0.7.1.1
simplegeneric                  0.8.1
six                            1.15.0
sklearn                        0.0
smmap                          3.0.5
sniffio                        1.3.0
snowballstemmer                2.0.0
SoundFile                      0.10.3.post1
soupsieve                      2.3.2.post1
SQLAlchemy                     1.4.41
starlette                      0.26.1
streamlit                      1.13.0
streamlit-image-comparison     0.0.3
tabulate                       0.8.3
tb-nightly                     1.15.0a20190801
tb-paddle                      0.3.6
tenacity                       8.0.1
tensorboard                    2.1.0
tensorboardX                   1.8
termcolor                      1.1.0
terminado                      0.15.0
terminaltables                 3.1.10
testpath                       0.4.2
threadpoolctl                  2.1.0
tinycss2                       1.1.1
tokenizers                     0.13.3
toml                           0.10.0
toolz                          0.12.0
torch                          1.13.1
tornado                        5.1.1
tqdm                           4.64.1
traitlets                      5.4.0
transformers                   4.27.1
typed-ast                      1.4.1
typing_extensions              4.3.0
tzdata                         2022.7
tzlocal                        4.2
uc-micro-py                    1.0.1
ujson                          1.35
urllib3                        1.25.11
uvicorn                        0.21.1
validators                     0.20.0
virtualenv                     16.7.9
visualdl                       2.4.0
watchdog                       2.2.0
wcwidth                        0.1.7
webencodings                   0.5.1
websocket-client               1.4.1
websockets                     10.4
Werkzeug                       0.16.0
whatthepatch                   1.0.2
wheel                          0.36.2
widgetsnbextension             3.5.2
wrapt                          1.12.1
xarray                         0.16.2
xgboost                        1.3.3
xlrd                           1.2.0
xxhash                         3.1.0
yapf                           0.26.0
yarl                           1.7.2
zipp                           3.8.1

### Anything else?

_No response_

[duzx16]

环境问题。应该是 PyTorch 编译时的 CUDA 版本和系统的 CUDA 版本不匹配。建议按照 PyTorch 官网 的说明用 conda 安装