Closed fatbringer closed 1 year ago
Is this a problem with my CUDA version ? Or is this a problem with selecting wrong version paddle to do pip install ?
Hi, please check whether libcudart.so.11.0
exists in your environment and is added to LD_LIBRARY_PATH
.And could you please provide the cuDNN version in your environment?
I have libcudart.so installed
whereis libcudart.so.11.0
libcudart.so.11.0: /usr/lib/x86_64-linux-gnu/libcudart.so.11.0
Inside /etc/ld.so.conf.d
# Multiarch support
/usr/local/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
Since the directory contains libcudart.so.11.0, sudo ldconfig should load it. Is that right?
Strangely, i am unable to find the cuDNN files, despite having nvidia-smi being able to run, and other frameworks such as pytorch work on my laptop
cat /usr/local/cuda/version.txt
cat: /usr/local/cuda/version.txt: No such file or directory
cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
cat: /usr/local/cuda/include/cudnn.h: No such file or directory
cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2
cat: /usr/include/cudnn.h: No such file or directory
The driver installation seems to be valid though
cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 470.199.02 Thu May 11 11:46:56 UTC 2023
GCC version: gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
Seems like i can import paddle now
But i have gotten another error when verifying the install with paddle.utils.run_check()
>>> import paddle
>>> paddle.utils.run_check()
Running verify PaddlePaddle program ...
I0714 09:49:56.078572 16365 interpretercore.cc:237] New Executor is Running.
W0714 09:49:56.079103 16365 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.4, Runtime API Version: 11.5
W0714 09:49:56.079572 16365 dynamic_loader.cc:303] The third-party dynamic library (libcudnn.so) that Paddle depends on is not configured correctly. (error code is /usr/local/cuda/lib64/libcudnn.so: cannot open shared object file: No such file or directory)
Suggestions:
1. Check if the third-party dynamic library (e.g. CUDA, CUDNN) is installed correctly and its version is matched with paddlepaddle you installed.
2. Configure third-party dynamic library environment variables as follows:
- Linux: set LD_LIBRARY_PATH by `export LD_LIBRARY_PATH=...`
- Windows: set PATH by `set PATH=XXX;
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 249, in run_check
_run_static_single(use_cuda, use_xpu)
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 146, in _run_static_single
exe.run(startup_prog)
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/executor.py", line 1392, in run
res = self._run_impl(
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/executor.py", line 1618, in _run_impl
ret = new_exe.run(
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/executor.py", line 654, in run
tensors = self._new_exe.run(
RuntimeError: In user code:
File "<stdin>", line 1, in <module>
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 249, in run_check
_run_static_single(use_cuda, use_xpu)
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 133, in _run_static_single
input, out, weight = _simple_network()
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 31, in _simple_network
weight = paddle.create_parameter(
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/tensor/creation.py", line 222, in create_parameter
return helper.create_parameter(
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/layer_helper_base.py", line 432, in create_parameter
self.startup_program.global_block().create_parameter(
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/framework.py", line 3935, in create_parameter
initializer(param, self)
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/nn/initializer/initializer.py", line 40, in __call__
return self.forward(param, block)
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/nn/initializer/constant.py", line 65, in forward
op = block.append_op(
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/framework.py", line 4013, in append_op
op = Operator(
File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/framework.py", line 2781, in __init__
for frame in traceback.extract_stack():
PreconditionNotMetError: Cannot load cudnn shared library. Cannot invoke method cudnnGetVersion.
[Hint: cudnn_dso_handle should not be null.] (at ../paddle/phi/backends/dynload/cudnn.cc:64)
[operator < fill_constant > error]
Seems like i can import paddle now
But i have gotten another error when verifying the install with paddle.utils.run_check()
>>> import paddle >>> paddle.utils.run_check() Running verify PaddlePaddle program ... I0714 09:49:56.078572 16365 interpretercore.cc:237] New Executor is Running. W0714 09:49:56.079103 16365 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.4, Runtime API Version: 11.5 W0714 09:49:56.079572 16365 dynamic_loader.cc:303] The third-party dynamic library (libcudnn.so) that Paddle depends on is not configured correctly. (error code is /usr/local/cuda/lib64/libcudnn.so: cannot open shared object file: No such file or directory) Suggestions: 1. Check if the third-party dynamic library (e.g. CUDA, CUDNN) is installed correctly and its version is matched with paddlepaddle you installed. 2. Configure third-party dynamic library environment variables as follows: - Linux: set LD_LIBRARY_PATH by `export LD_LIBRARY_PATH=...` - Windows: set PATH by `set PATH=XXX; Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 249, in run_check _run_static_single(use_cuda, use_xpu) File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 146, in _run_static_single exe.run(startup_prog) File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/executor.py", line 1392, in run res = self._run_impl( File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/executor.py", line 1618, in _run_impl ret = new_exe.run( File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/executor.py", line 654, in run tensors = self._new_exe.run( RuntimeError: In user code: File "<stdin>", line 1, in <module> File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 249, in run_check _run_static_single(use_cuda, use_xpu) File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 133, in _run_static_single input, out, weight = _simple_network() File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/utils/install_check.py", line 31, in _simple_network weight = paddle.create_parameter( File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/tensor/creation.py", line 222, in create_parameter return helper.create_parameter( File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/layer_helper_base.py", line 432, in create_parameter self.startup_program.global_block().create_parameter( File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/framework.py", line 3935, in create_parameter initializer(param, self) File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/nn/initializer/initializer.py", line 40, in __call__ return self.forward(param, block) File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/nn/initializer/constant.py", line 65, in forward op = block.append_op( File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/framework.py", line 4013, in append_op op = Operator( File "/home/aevas/Desktop/pyenvs/newpy39/lib/python3.9/site-packages/paddle/fluid/framework.py", line 2781, in __init__ for frame in traceback.extract_stack(): PreconditionNotMetError: Cannot load cudnn shared library. Cannot invoke method cudnnGetVersion. [Hint: cudnn_dso_handle should not be null.] (at ../paddle/phi/backends/dynload/cudnn.cc:64) [operator < fill_constant > error]
Hi, it seems that you have not installed cudnn properly, maybe you should try to reinstall it.
@USTCKAY ok i have reinstalled my cuda and cudnn and it seems to be ok!
>>> import paddle
>>> paddle.utils.run_check()
Running verify PaddlePaddle program ...
I0717 09:40:58.291146 12137 interpretercore.cc:237] New Executor is Running.
W0717 09:40:58.292129 12137 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 11.7
W0717 09:40:58.293215 12137 gpu_resources.cc:149] device: 0, cuDNN Version: 8.9.
I0717 09:41:00.817838 12137 interpreter_util.cc:518] Standalone Executor is Used.
PaddlePaddle works well on 1 GPU.
PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.
>>> exit()
In case people need, these were helpful
@USTCKAY ok i have reinstalled my cuda and cudnn and it seems to be ok!
>>> import paddle >>> paddle.utils.run_check() Running verify PaddlePaddle program ... I0717 09:40:58.291146 12137 interpretercore.cc:237] New Executor is Running. W0717 09:40:58.292129 12137 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 11.7 W0717 09:40:58.293215 12137 gpu_resources.cc:149] device: 0, cuDNN Version: 8.9. I0717 09:41:00.817838 12137 interpreter_util.cc:518] Standalone Executor is Used. PaddlePaddle works well on 1 GPU. PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now. >>> exit()
In case people need, these were helpful
Congrats!
Solved
sudo apt-get install libpython3.8-dev #python3.8
lack of software packs
bug描述 Describe the Bug
Importing paddle on my python3.9 virtual environment gets me an error, i am unable to import paddle due to the existence of a certain file. I have tried paddlepaddle-gpu 2.4.2 and 2.50, both versions do not work.
I have consulted this other issue here https://github.com/PaddlePaddle/Paddle/issues/48681, https://github.com/PaddlePaddle/PaddleOCR/issues/1304 but it does not work for me.
i have ran
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/aevas/Desktop/pyenvs/newpy39/lib
I have ran
pip install paddlepaddle-gpu
This is my nvidia-smi output
其他补充信息 Additional Supplementary Information
I tried running the following in a jupyter notebook in my virtual environment