PaddlePaddle / Paddle

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice (『飞桨』核心框架,深度学习&机器学习高性能单机、分布式训练和跨平台部署)
http://www.paddlepaddle.org/
Apache License 2.0
21.66k stars 5.44k forks source link

CUBLAS_STATUS_NOT_SUPPORTED when install paddle-2.5.2.post112 #64056

Open xiaojinglu opened 1 week ago

xiaojinglu commented 1 week ago

问题描述 Issue Description

paddle安装后出现CUBLAS_STATUS_NOT_SUPPORTED错误

版本&环境信息 Version & Environment Information

环境: nvidia-smi | NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4 |

安装脚本

python -m pip install paddlepaddle-gpu==2.5.2.post112 -f https://www.paddlepaddle.org.cn/whl/windows/mkl/avx/stable.html

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Sun_Feb_14_21:12:58_PST_2021 Cuda compilation tools, release 11.2, V11.2.152 Build cuda_11.2.r11.2/compiler.29618528_0

python

Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux

lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal

代码检查运行报错 ` (chatchat-gpu-cudnn) app.e0015950@s0pgpuap05:~/star/pyprojects/ChatDocs-v0.4-test-newest/ChatDocs_query_newest$ python Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import paddle pa>>> paddle.utils.run_check() Running verify PaddlePaddle program ... I0506 17:14:56.319113 603358 interpretercore.cc:237] New Executor is Running. W0506 17:14:56.320781 603358 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.4, Runtime API Version: 11.2 W0506 17:14:56.323613 603358 gpu_resources.cc:149] device: 0, cuDNN Version: 8.2. Traceback (most recent call last): File "", line 1, in File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/utils/install_check.py", line 249, in run_check _run_static_single(use_cuda, use_xpu) File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/utils/install_check.py", line 147, in _run_static_single exe.run( File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/fluid/executor.py", line 1392, in run res = self._run_impl( File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/fluid/executor.py", line 1618, in _run_impl ret = new_exe.run( File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/fluid/executor.py", line 654, in run tensors = self._new_exe.run( OSError: In user code:

File "<stdin>", line 1, in <module>

File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/utils/install_check.py", line 249, in run_check
  _run_static_single(use_cuda, use_xpu)
File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/utils/install_check.py", line 133, in _run_static_single
  input, out, weight = _simple_network()
File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/utils/install_check.py", line 37, in _simple_network
  linear_out = paddle.nn.functional.linear(x=input, weight=weight, bias=bias)
File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/nn/functional/common.py", line 1860, in linear
  helper.append_op(
File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/fluid/layer_helper.py", line 45, in append_op
  return self.main_program.current_block().append_op(*args, **kwargs)
File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/fluid/framework.py", line 4013, in append_op
  op = Operator(
File "/home/app.e0015950/miniconda3/envs/chatchat-gpu-cudnn/lib/python3.10/site-packages/paddle/fluid/framework.py", line 2781, in __init__
  for frame in traceback.extract_stack():

ExternalError: CUBLAS error(15).
  [Hint: 'CUBLAS_STATUS_NOT_SUPPORTED'.  The functionality requested is not supported ] (at ../paddle/phi/kernels/funcs/blas/blas_impl.cu.h:41)
  [operator < matmul_v2 > error]

` ExternalError: CUBLAS error(15). [Hint: 'CUBLAS_STATUS_NOT_SUPPORTED'. The functionality requested is not supported ] (at ../paddle/phi/kernels/funcs/blas/blas_impl.cu.h:41 有谁可以帮忙解决下么

lxd-cumt commented 1 week ago

似乎是环境的问题,可以尝试一下从源码编译一下,