使用华为NPU推理pp-yoloe-r模型报错，提示cann版本不匹配

mandicc commented 1 month ago

Host 环境：Ubuntu 18.04 华为NPU：A300Ipro驱动固件已装，CANN版本为5.1.RC1.alpha001

操作过程

1、使用如下命令配置paddle-lite环境： ./lite/tools/build_linux.sh --arch=x86 --with_extra=ON --with_log=ON --with_exception=ON --with_nnadapter=ON --nnadapter_with_huawei_ascend_npu=ON --nnadapter_huawei_ascend_npu_sdk_root=/usr/local/Ascend/ascend-toolkit/latest full_publish

2、之后安装Fastdeploy环境并安装whl git clone https://github.com/PaddlePaddle/FastDeploy.git cd FastDeploy/python export WITH_ASCEND=ON export ENABLE_VISION=ON python setup.py build python setup.py bdist_wheel

3、进入如下路径，复制准备好的paddle模型和测试图片，模型为pp-yolor-e 路径：FastDeploy/examples/vision/detection/paddledetection/python/

4、进行推理 python infer_ppyoloe_r.py --model_dir ppyoloe_r_crn_l_3x_dota --image P08611.01154___824.png --device ascend

部分报错信息如下：

[INFO] fastdeploy/runtime/runtime.cc(354)::CreateLiteBackend Runtime initialized with Backend::PDLITE in Device::ASCEND. 2222 [W 5/ 9 20:23:15.461 .../src/driver/huawei_ascend_npu/utility.cc:57 InitializeAscendCL] CANN version mismatch. The build version is 5.1.0, but the current environment version is 5.1.2. [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:41 Context] properties: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:66 Context] selected device ids: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:68 Context] 0 [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:78 Context] profiling path: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:88 Context] dump model path: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:98 Context] precision mode: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:120 Context] op select impl mode: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:130 Context] op type list for impl mode: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:140 Context] enable compressw weight: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:150 Context] auto tune mode: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:160 Context] enable dynamic shape range: [I 5/ 9 20:23:15.495 ...r/src/driver/huawei_ascend_npu/engine.cc:176 Context] initial buffer length of dynamic shape range: -1 [W 5/ 9 20:23:15.495 ...ter/nnadapter/src/runtime/compilation.cc:334 Finish] Warning: Failed to create a program, No model and cache is provided. [W 5/ 9 20:23:15.495 ...le-Lite/lite/kernels/nnadapter/engine.cc:149 LoadFromCache] Warning: Build model failed(3) ! [W 5/ 9 20:23:15.512 ...nnadapter/nnadapter/src/runtime/model.cc:86 GetSupportedOperations] Warning: Failed to get the supported operations for device 'huawei_ascend_npu', because the HAL interface 'validate_program' is not implemented! [W 5/ 9 20:23:15.512 ...kernels/nnadapter/converter/converter.cc:171 Apply] Warning: Failed to get the supported operations for the selected devices, one or more of the selected devices are not supported! [I 5/ 9 20:23:15.512 ...r/src/driver/huawei_ascend_npu/driver.cc:70 CreateProgram] Create program for huawei_ascend_npu. [F 5/ 9 20:23:58.112 ...driver/huawei_ascend_npu/model_client.cc:54 InitAclClientEnv] Check failed: (reinterpret_cast(aclrtSetDevice(deviceid)) == ACL_ERROR_NONE): 507033!==0 507033 Unknown ACL error code(507033) [F 5/ 9 20:23:58.112 ...driver/huawei_ascend_npu/model_client.cc:54 InitAclClientEnv] Check failed: (reinterpret_cast(aclrtSetDevice(deviceid)) == ACL_ERROR_NONE): 507033!==0 507033 Unknown ACL error code(507033)

[F 5/ 9 20:23:58.125 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found. [F 5/ 9 20:23:58.125 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

terminate called after throwing an instance of 'nnadapter::logging::Exception' what(): NNAdapter C++ Exception: [F 5/ 9 20:23:58.125 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

Aborted (core dumped) root@hitsz-NF5280M5:/home/hitsz/PP-Yoloe-R/FastDeploy/examples/vision/detection/paddledetection/python# /usr/local/python3.7.5/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 96 leaked semaphores to clean up at shutdown len(cache))

csy0225 commented 1 month ago

检查一下环境是否安装好，看报错 log 应该是没有找到 device，其他的报错都是 warning，不影响运行 [F 5/ 9 20:23:58.112 ...driver/huawei_ascend_npu/model_client.cc:54 InitAclClientEnv] Check failed: (reinterpret_cast(aclrtSetDevice(deviceid)) == ACL_ERROR_NONE): 507033!==0 507033 Unknown ACL error code(507033)

mandicc commented 1 month ago

好的谢谢您，请问这个device报错是华为那边环境问题还是paddlelite或者fastdeploy的环境问题呢，我好针对排查

csy0225 commented 1 month ago

这个应该是你本身昇腾环境安装的问题吧，看下驱动安装。

mandicc commented 1 month ago

您好，我进行了昇腾的环境的健康检查和兼容性检查，调整了一些不兼容的地方，也请了华为的工程师帮忙查看，当前npu环境运行正常，环境版本如下： d530ec67d3ae5126071fbb21ce0ac73 我使用pp-yoloe-r再次进行推理，仍然报出相同的错误。我是否需要重装paddlelite或者fastdeploy，或者给您提供报错的日志信息或者其他信息。项目即将验收，请您继续提供帮助，十分感谢！

csy0225 commented 1 month ago

您好，目前 cann 版本8.x 我们这边没有适配，最高验证过6.x，所以可能需要您自己适配一下了。

csy0225 commented 1 month ago

您调试的时候也可以打开 ascend 的日志，看下 log 。

csy0225 commented 1 month ago

确认下你的 fastdeploy 调用的是编译出来的 Paddle-Lite 么？是否需要设置一下 export LD_LIBRARY_PATH? 感觉像是版本不对，以至于没有找到 so。

csy0225 commented 1 month ago

您可以用一下我们官方 demo 尝试跑一下，排除 fastdeploy 的干扰，如果官方 Demo 没问题，在使用 fastdeploy 进行集成。

mandicc commented 1 month ago

您好，我下载了PaddleLite-generic-demo.tar.gz，使用./run.sh mobilenet_v1_fp32_224 imagenet_224.txt test linux amd64 huawei_ascend_npu运行了一个demo，程序报错并退出。

我的paddlelite当时是从github上下载后使用命令 ./lite/tools/build_linux.sh --arch=x86 --with_extra=ON --with_log=ON --with_exception=ON --with_nnadapter=ON --nnadapter_with_huawei_ascend_npu=ON --nnadapter_huawei_ascend_npu_sdk_root=/usr/local/Ascend/ascend-toolkit/latest full_publish 是直接装在物理机上的，并未使用容器，请问paddlelite如何卸载，当时我的昇腾存在系统兼容性问题，我准备卸载后重新安装paddlelite，或者使用官方建议的容器进行安装

csy0225 commented 1 month ago

用demo的话，你不是用python跑的，所以手动指定一下LD_LIBRARY_PATH 路径就好了，配置一下run.sh的脚本即可，无需卸载。

mandicc commented 1 month ago

您指的LD_LIBRARY_PATH 路径是哪条路径，run.sh中的哪部分需要更改 run.sh与昇腾有关的部分如下：

if [[ "$NNADAPTER_DEVICE_NAMES" =~ "huawei_ascend_npu" ]]; then
  HUAWEI_ASCEND_TOOLKIT_HOME="/usr/local/Ascend/ascend-toolkit/latest"
  if [ "$TARGET_OS" == "linux" ]; then
    if [[ "$TARGET_ABI" != "arm64" && "$TARGET_ABI" != "amd64" ]]; then
      echo "Unknown OS $TARGET_OS, only supports 'arm64' or 'amd64' for Huawei Ascend NPU."
      exit -1
    fi
  else
    echo "Unknown OS $TARGET_OS, only supports 'linux' for Huawei Ascend NPU."
    exit -1
  fi
  if [[ ! "$NNADAPTER_CONTEXT_PROPERTIES" =~ "HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS" ]]; then
    NNADAPTER_CONTEXT_PROPERTIES="HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS=0;${NNADAPTER_CONTEXT_PROPERTIES}"
  fi
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/stub:$HUAWEI_ASCEND_TOOLKIT_HOME/acllib/lib64:$HUAWEI_ASCEND_TOOLKIT_HOME/atc/lib64:$HUAWEI_ASCEND_TOOLKIT_HOME/opp/op_proto/built-in
  export PYTHONPATH=$PYTHONPATH:$HUAWEI_ASCEND_TOOLKIT_HOME/fwkacllib/python/site-packages:$HUAWEI_ASCEND_TOOLKIT_HOME/acllib/python/site-packages:$HUAWEI_ASCEND_TOOLKIT_HOME/toolkit/python/site-packages:$HUAWEI_ASCEND_TOOLKIT_HOME/atc/python/site-packages:$HUAWEI_ASCEND_TOOLKIT_HOME/pyACL/python/site-packages/acl
  export PATH=$PATH:$HUAWEI_ASCEND_TOOLKIT_HOME/atc/ccec_compiler/bin:${HUAWEI_ASCEND_TOOLKIT_HOME}/acllib/bin:$HUAWEI_ASCEND_TOOLKIT_HOME/atc/bin
  export ASCEND_AICPU_PATH=$HUAWEI_ASCEND_TOOLKIT_HOME
  export ASCEND_OPP_PATH=$HUAWEI_ASCEND_TOOLKIT_HOME/opp
  export TOOLCHAIN_HOME=$HUAWEI_ASCEND_TOOLKIT_HOME/toolkit
  export ASCEND_SLOG_PRINT_TO_STDOUT=1
  export ASCEND_GLOBAL_LOG_LEVEL=3
fi

demo部分报错信息如下，前几行和fastdeploy的报错相同：

[F  5/10 20:28:31. 79 .../src/driver/huawei_ascend_npu/utility.cc:315 BuildOMModelToBuffer] Check failed: (reinterpret_cast<ge::graphStatus>(aclgrphBuildModel(ir_graph, options, om_buffer)) == ge::GRAPH_SUCCESS): 1343266818!==0 1343266818 Unknown ATC error code(1343266818)
[F  5/10 20:28:31. 79 .../src/driver/huawei_ascend_npu/utility.cc:315 BuildOMModelToBuffer] Check failed: (reinterpret_cast<ge::graphStatus>(aclgrphBuildModel(ir_graph, options, om_buffer)) == ge::GRAPH_SUCCESS): 1343266818!==0 1343266818 Unknown ATC error code(1343266818)

[F  5/10 20:28:31.258 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.
[F  5/10 20:28:31.258 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

terminate called after throwing an instance of 'nnadapter::logging::Exception'
  what():  NNAdapter C++ Exception:
[F  5/10 20:28:31.258 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

./run.sh: line 177: 24022 Aborted                 (core dumped) ./$BUILD_DIR/demo ../assets/models/$MODEL_NAME ../assets/configs/$CONFIG_NAME ../assets/datasets/$DATASET_NAME $NNADAPTER_DEVICE_NAMES "$NNADAPTER_CONTEXT_PROPERTIES" $NNADAPTER_MODEL_CACHE_DIR $NNADAPTER_MODEL_CACHE_TOKEN $NNADAPTER_SUBGRAPH_PARTITION_CONFIG_PATH $NNADAPTER_MIXED_PRECISION_QUANTIZATION_CONFIG_PATH
root@hitsz-NF5280M5:/home/hitsz/PP-Yoloe-R/PaddleLite-generic-demo/image_classification_demo/shell# [ERROR] TBE(24150,python3):2024-05-10-20:28:31.489.433 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
[ERROR] TBE(24151,python3):2024-05-10-20:28:31.489.388 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
[ERROR] TBE(24149,python3):2024-05-10-20:28:31.489.469 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
[ERROR] TBE(24153,python3):2024-05-10-20:28:31.489.467 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
[ERROR] TBE(24154,python3):2024-05-10-20:28:31.489.474 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
[ERROR] TBE(24148,python3):2024-05-10-20:28:31.489.516 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
[ERROR] TBE(24152,python3):2024-05-10-20:28:31.489.596 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
[ERROR] TBE(24147,python3):2024-05-10-20:28:31.489.591 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:61][repository_manager] Subprocess[task_distribute] raise error[]
Process ForkServerPoolWorker-5:
Process ForkServerPoolWorker-6:
Process ForkServerPoolWorker-4:
Process ForkServerPoolWorker-9:
Process ForkServerPoolWorker-8:
Process ForkServerPoolWorker-3:
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
Traceback (most recent call last):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

During handling of the above exception, another exception occurred:

  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
BrokenPipeError: [Errno 32] Broken pipe
BrokenPipeError: [Errno 32] Broken pipe
Process ForkServerPoolWorker-2:
Process ForkServerPoolWorker-7:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 127, in worker
    put((job, i, result))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)

During handling of the above exception, another exception occurred:

  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
Traceback (most recent call last):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/pool.py", line 132, in worker
    put((job, i, (False, wrapped)))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/queues.py", line 364, in put
    self._writer.send_bytes(obj)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/usr/local/python3.7.5/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 39 leaked semaphores to clean up at shutdown
  len(cache))

csy0225 commented 1 month ago

如果你ascend 那部分环境变量设置正确的话，还出现错误那就是需要单独适配了，需要开启ascend相关日志看下是什么原因

mandicc commented 1 month ago

您好，这个问题看起来很难解决，有可能是难以兼容8.0rc1的华为软件，我准备重新开始，将paddlelite卸载干净后把昇腾的软件降到6.0.RC1再测试demo，有几个问题向您请教 ①请问您知道paddlelite的卸载方式吗，官网好像没有 ②此外通过在github上下载paddlelite后使用./lite/tools/build_linux.sh --arch=x86 --with_extra=ON --with_log=ON --with_exception=ON --with_nnadapter=ON --nnadapter_with_huawei_ascend_npu=ON --nnadapter_huawei_ascend_npu_sdk_root=/usr/local/Ascend/ascend-toolkit/latest full_publish直接在物理机上安装paddlelite是否合理 ③模型的部署是否要求下载昇腾的nnae而非nnrt ④如果只运行PaddleLite-generic-demo.tar.gz中的demo是否需要以完成②步为前提

csy0225 commented 1 month ago

1.paddlelite如果不是python编译的话，不需要卸载，运行时通过指定LD_LIBRARY_PATH 路径后生效，如果是python安装的，可以使用pip unistall paddlelite 进行卸载。

paddlelite是以动态库的形式运行，所以只要运行时，设置正确的路径即可，物理机 docker 均可。
我们这边也是使用的nnrt,可以参考下面
这个如果装的环境和paddlelite预编译的环境不一样可能需要重新编译，但是如果您安装好了环境的话，可以先使用我们的预编译库先试着跑一下，有问题在重新编译。

mandicc commented 1 month ago

我注意到您提到环境变量的相关问题，请您帮助查看环境变量是否包含您所提到的paddlelite正确路径，我的paddlelite未经过python编译

PaddlePaddle / Paddle-Lite

使用华为NPU推理pp-yoloe-r模型报错，提示cann版本不匹配 #10514

操作过程

部分报错信息如下：