PaddlePaddle / PaddleNLP

👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
https://paddlenlp.readthedocs.io
Apache License 2.0
11.96k stars 2.91k forks source link

[Bug]: undefined symbol: _ZN6paddle17OpMetaInfoBuilderC1EOSsm #8843

Open Chuge0335 opened 1 month ago

Chuge0335 commented 1 month ago

软件环境

paddle-bfloat              0.1.7
paddle2onnx                1.2.1
paddlefsl                  1.1.0
paddlehub                  2.4.0
paddlemix                  0.1.0               
paddlenlp                  3.0.0b0.post20240728
paddlepaddle-gpu           2.6.1.post112
paddleslim                 2.6.0

重复问题

错误描述

在编译csrc 算子时报错未定义的符号:_ZN6paddle17OpMetaInfoBuilderC1EOSsm
在相关问题中并未得到解决:
[按paddle教程自定义算子时链接库找不到](https://aistudio.baidu.com/paddle/forum/topic/show/1729464)
[#8222](https://github.com/PaddlePaddle/PaddleDetection/issues/8222)
[#9000](https://github.com/PaddlePaddle/PaddleDetection/issues/9000)
[#4045](https://github.com/PaddlePaddle/PaddleNLP/issues/4045)
[#43041](https://github.com/PaddlePaddle/Paddle/issues/43041)
其中[#8222]提到更换gcc版本,尝试过切换gcc 11.2 和gcc 8.5 无效

### 报错log输出

[2024-07-31 13:40:37,079] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/write_cache_kv.cu.o -> build/paddlenlp_ops/bdist.linux-x86_64/egg        
[2024-07-31 13:40:37,082] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/paddlenlp_ops.so -> build/paddlenlp_ops/bdist.linux-x86_64/egg           
[2024-07-31 13:40:37,099] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/dequant_int8.cu.o -> build/paddlenlp_ops/bdist.linux-x86_64/egg          
[2024-07-31 13:40:37,103] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/transpose_removing_padding.cu.o -> build/paddlenlp_ops/bdist.linux-x86_64
/egg                                                                                                                                                                                     
[2024-07-31 13:40:37,108] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/save_with_output_msg.o -> build/paddlenlp_ops/bdist.linux-x86_64/egg     
[2024-07-31 13:40:37,183] [    INFO] bdist_egg.py:177 - creating stub loader for paddlenlp_ops.so                                                                                        
Traceback (most recent call last):                                                                                                                                                       
  File "/mnt/petrelfs/taitan/projects/PaddleMIX/PaddleNLP/csrc/setup_cuda.py", line 53, in <module>                                                                                     
    setup(                                                                                                                                                                               
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/paddle/utils/cpp_extension/cpp_extension.py", line 236, in setup                                        
    setuptools.setup(**attr)                                                                                                                                                             
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/__init__.py", line 104, in setup                                                             
    return distutils.core.setup(**attrs)                                                                                                                                                 
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 184, in setup
    return run_commands(dist)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
    dist.run_commands()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/dist.py", line 967, in run_command    super().run_command(command)  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/command/install.py", line 87, in run
    self.do_egg_install()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/command/install.py", line 139, in do_egg_install
    self.run_command('bdist_egg')
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/dist.py", line 967, in run_command
    super().run_command(command)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/command/bdist_egg.py", line 179, in run
    write_stub(os.path.basename(ext_name), pyfile)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/paddle/utils/cpp_extension/extension_utils.py", line 222, in custom_write_stub
    new_custom_ops = load_op_meta_info_and_register_op(so_path)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/paddle/utils/cpp_extension/extension_utils.py", line 160, in load_op_meta_info_and_register_op
    core.load_op_meta_info_and_register_op(lib_filename)
RuntimeError: (PreconditionNotMet) The third-party dynamic library (/mnt/petrelfs/taitan/projects/PaddleMIX/PaddleNLP/csrc/build/paddlenlp_ops/lib.linux-x86_64-cpython-39/paddlenlp_ops
.so) that Paddle depends on is not configured correctly. (error code is /mnt/petrelfs/taitan/projects/PaddleMIX/PaddleNLP/csrc/build/paddlenlp_ops/lib.linux-x86_64-cpython-39/paddlenlp
_ops.so: undefined symbol: _ZN6paddle17OpMetaInfoBuilderC1EOSsm)
  Suggestions:
  1. Check if the third-party dynamic library (e.g. CUDA, CUDNN) is installed correctly and its version is matched with paddlepaddle you installed.
  2. Configure third-party dynamic library environment variables as follows:
  - Linux: set LD_LIBRARY_PATH by `export LD_LIBRARY_PATH=...`
  - Windows: set PATH by `set PATH=XXX; (at ../paddle/phi/backends/dynload/dynamic_loader.cc:312)


### 稳定复现步骤 & 代码

Ubuntu 20.04.6 LTS
A100 80G
cuda 11.2
gcc 8.5.0
python=3.9.19

cudnn
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 1
DrownFish19 commented 1 month ago

这个问题由系统环境引起,可能不同环境有不同原因,此处给出一个检查方案