Open adamzhg opened 2 months ago
换回预置的lib和头文件之后,也是cpu可以跑通,npu报错: ository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/route.py:62][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1436413,python3):2024-04-25-13:04:33.029.899 [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/route.py:62][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1436409,python3):2024-04-25-13:04:33.029.900 [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/route.py:62][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1436415,python3):2024-04-25-13:04:33.029.900 [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/route.py:62][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1436412,python3):2024-04-25-13:04:33.029.952 [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/route.py:62][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1436410,python3):2024-04-25-13:04:33.029.947 [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/route.py:62][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1436414,python3):2024-04-25-13:04:33.030.022 [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/compiler/python/site-packages/tbe/common/repository_manager/route.py:62][repository_manager] Subprocess[task_distribute] raise error[] Process ForkServerPoolWorker-2: Process ForkServerPoolWorker-6: Process ForkServerPoolWorker-8: Process ForkServerPoolWorker-9: Process ForkServerPoolWorker-4: Process ForkServerPoolWorker-7: Process ForkServerPoolWorker-5: Process ForkServerPoolWorker-3: Traceback (most recent call last): Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last):
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) Traceback (most recent call last): Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result))
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/usr/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(self._args, **self._kwargs) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf)
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf)
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run()
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run()
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/usr/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe /usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 97 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '
目前已经验证过的驱动版本是 1.0.9.alpha,CANN版本是 5.0.2.alpha005,可以检查下本地版本是否一致
新版本的驱动和CANN版本目前没有验证过,可能需要一定的适配代码修改,可以自行尝试下~
版本、环境信息: 1)Paddle Lite 版本:git clone的最新开发代码 2)Host 环境:Ubuntu 20.04 3)运行设备环境:华为鲲鹏服务器+Atlas 300I(3010,昇腾310) 4)CANN 7.0
编译/安装方式信息: 参考https://www.paddlepaddle.org.cn/lite/develop/demo_guides/huawei_ascend_npu.html 进行安装
错误信息: 参考官网编译了paddle-lite和进行了include、lib替换,cpu运行没有问题,npu运行报错: ./run.sh mobilenet_v1_fp32_224 imagenet_224.txt test linux arm64,cpu运行没有问题,正常打出结果; ./run.sh mobilenet_v1_fp32_224 imagenet_224.txt test linux arm64 huawei_ascend_npu,npu运行报错: [F 4/24 17:22:10.685 ...nadapter/nnadapter/src/runtime/device.cc:529 Find] Failed to load the nnadapter device HAL library for 'huawei_ascend_npu' from libhuawei_ascend_npu.so, ../../libs/PaddleLite/linux/arm64/lib/huawei_ascend_npu/libhuawei_ascend_npu.so: undefined symbol: PyExc_ValueError [F 4/24 17:22:10.685 ...nadapter/nnadapter/src/runtime/device.cc:529 Find] Failed to load the nnadapter device HAL library for 'huawei_ascend_npu' from libhuawei_ascend_npu.so, ../../libs/PaddleLite/linux/arm64/lib/huawei_ascend_npu/libhuawei_ascend_npu.so: undefined symbol: PyExc_ValueError