PaddlePaddle / PaddleX

Low-code development tool based on PaddlePaddle(飞桨低代码开发工具)
Apache License 2.0
4.74k stars 932 forks source link

OSError: (External) Cuda error(3), initialization error. #1068

Open zzcqinag opened 3 years ago

zzcqinag commented 3 years ago

问题类型:模型训练

PaddleX版本v2.0.0rc1 cuda版本10.1 cudnn版本7.6.5 显卡是titian rtx 导入数据集然后点训练就报错 问题描述
This log file path is /home/zzc/paddlex_workspace/projects/P0002/T0007/err.log 注意:标志为WARNING/INFO类的仅为警告或提示类信息,非错误信息 /home/zzc/Downloads/paddlex_gui_ubuntu1804_amd64_v2.0.0rc1/PaddleX/paddle/fluid/framework.py:301: UserWarning: You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default. /home/zzc/Downloads/paddlex_gui_ubuntu1804_amd64_v2.0.0rc1/PaddleX/paddle/distributed/parallel.py:120: UserWarning: Currently not a parallel execution environment, paddle.distributed.init_parallel_env will not do anything. /home/zzc/Downloads/paddlex_gui_ubuntu1804_amd64_v2.0.0rc1/PaddleX/paddle/tensor/creation.py:125: DeprecationWarning: np.object is a deprecated alias for the builtin object. To silence this warning, use object by itself. Doing this will not modify any behavior and is safe. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations Process Process-1:2: Traceback (most recent call last): File "multiprocessing/process.py", line 297, in _bootstrap File "multiprocessing/process.py", line 99, in run File "paddlexui/pms/model_tasks/tasks.py", line 58, in _call_paddlex_train File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/init.py", line 20, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/cv/init.py", line 15, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/cv/models/init.py", line 17, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/cv/models/detector.py", line 25, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/init.py", line 15, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/data/init.py", line 16, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/data/transform/init.py", line 15, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/data/transform/operators.py", line 41, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/modeling/init.py", line 22, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/modeling/proposal_generator/init.py", line 1, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/modeling/proposal_generator/rpn_head.py", line 58, in File "site-packages/paddlex/ppdet/modeling/proposal_generator/rpn_head.py", line 74, in RPNHead File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 57, in init File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 83, in _calculate_anchors File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 83, in File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 76, in generate_cell_anchors File "", line 2, in to_tensor File "site-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl File "site-packages/paddle/fluid/framework.py", line 227, in impl File "site-packages/paddle/tensor/creation.py", line 160, in to_tensor OSError: (External) Cuda error(3), initialization error. [Advise: Please search for the error code(3) on website( https://docs.nvidia.com/cuda/archive/9.0/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038 ) to get Nvidia's official solution about CUDA Error.] (at /paddle/paddle/fluid/platform/gpu_info.cc:99)

FlyingQianMM commented 3 years ago

2.0版本的GUI需要CUDA11.2, CUDNN8才可以使用

GUI(CUDA11.2/CUDNN8): windows: https://bj.bcebos.com/paddlex/PaddleX_30X0/PaddleX2.0.0rc2_windows10.zip ubuntu: https://bj.bcebos.com/paddlex/2.0//paddlex_gui_ubuntu1804_amd64_v2.0.0rc2.tar.gz

jack00000 commented 9 months ago

问题类型:模型训练

PaddleX版本v2.0.0rc1 cuda版本10.1 cudnn版本7.6.5 显卡是titian rtx 导入数据集然后点训练就报错 问题描述 This log file path is /home/zzc/paddlex_workspace/projects/P0002/T0007/err.log 注意:标志为WARNING/INFO类的仅为警告或提示类信息,非错误信息 /home/zzc/Downloads/paddlex_gui_ubuntu1804_amd64_v2.0.0rc1/PaddleX/paddle/fluid/framework.py:301: UserWarning: You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default. /home/zzc/Downloads/paddlex_gui_ubuntu1804_amd64_v2.0.0rc1/PaddleX/paddle/distributed/parallel.py:120: UserWarning: Currently not a parallel execution environment, paddle.distributed.init_parallel_env will not do anything. /home/zzc/Downloads/paddlex_gui_ubuntu1804_amd64_v2.0.0rc1/PaddleX/paddle/tensor/creation.py:125: DeprecationWarning: np.object is a deprecated alias for the builtin object. To silence this warning, use object by itself. Doing this will not modify any behavior and is safe. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations Process Process-1:2: Traceback (most recent call last): File "multiprocessing/process.py", line 297, in _bootstrap File "multiprocessing/process.py", line 99, in run File "paddlexui/pms/model_tasks/tasks.py", line 58, in _call_paddlex_train File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/init.py", line 20, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/cv/init.py", line 15, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/cv/models/init.py", line 17, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/cv/models/detector.py", line 25, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/init.py", line 15, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/data/init.py", line 16, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/data/transform/init.py", line 15, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/data/transform/operators.py", line 41, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/modeling/init.py", line 22, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/modeling/proposal_generator/init.py", line 1, in File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "/home/paddlex/.conda/envs/paddlex30X0/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module File "site-packages/paddlex/ppdet/modeling/proposal_generator/rpn_head.py", line 58, in File "site-packages/paddlex/ppdet/modeling/proposal_generator/rpn_head.py", line 74, in RPNHead File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 57, in init File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 83, in _calculate_anchors File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 83, in File "site-packages/paddlex/ppdet/modeling/proposal_generator/anchor_generator.py", line 76, in generate_cell_anchors File "", line 2, in to_tensor File "site-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl File "site-packages/paddle/fluid/framework.py", line 227, in impl File "site-packages/paddle/tensor/creation.py", line 160, in to_tensor OSError: (External) Cuda error(3), initialization error. [Advise: Please search for the error code(3) on website( https://docs.nvidia.com/cuda/archive/9.0/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038 ) to get Nvidia's official solution about CUDA Error.] (at /paddle/paddle/fluid/platform/gpu_info.cc:99)

解决拉没有 如何解决的