search

PaddlePaddle / Paddle-Lite

PaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge (飞桨高性能深度学习端侧推理引擎）
https://www.paddlepaddle.org.cn/lite
Apache License 2.0
6.96k stars 1.61k forks source link

昇腾310P3无法使用dynamic shape #10583

Open ly303550688 opened 6 days ago

ly303550688 commented 6 days ago

模型是paddlenlp文本纠错，下边是config代码，去掉动态shape可以正常运行

       config = CxxConfig()
        current_dir = os.getcwd()
        config.set_model_file("model/model.pdmodel")
        config.set_param_file("model/model.pdiparams")
        places = [
            Place(TargetType.NNAdapter, PrecisionType.INT8),
            Place(TargetType.NNAdapter, PrecisionType.FP16),
            Place(TargetType.ARM, PrecisionType.INT8),
            Place(TargetType.ARM, PrecisionType.FP32),
        ]
        config.set_nnadapter_device_names(["huawei_ascend_npu"])
        config.set_nnadapter_dynamic_shape_info(
            {"input_ids": [[-1, -1]], "pinyin_ids": [[-1, -1]]}
        )
        config.set_nnadapter_context_properties(
            "HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS=0;"
            + f"HUAWEI_ASCEND_NPU_DUMP_MODEL_FILE_PATH={current_dir}/model/cache/;"
            + "HUAWEI_ASCEND_NPU_ENABLE_DYNAMIC_SHAPE_RANGE=true;"
            + "HUAWEI_ASCEND_NPU_PRECISION_MODE=allow_fp32_to_fp16;"
            + "HUAWEI_ASCEND_NPU_OP_SELECT_IMPL_MODE=high_precision_for_all;"
        )
        config.set_nnadapter_model_cache_dir(
            os.path.join(current_dir, "model", "cache")
        )
        config.set_valid_places(places)
        self.predictor = create_paddle_predictor(config)
        self.predictor.save_optimized_pb_model(config.nnadapter_model_cache_dir())

报错信息： [W 10/31 23:18: 0.423 .../src/driver/huawei_ascend_npu/utility.cc:57 InitializeAscendCL] CANN version mismatch. The build version is 0.0.0, but the current environment version is 8.0.1. [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:42 Context] properties: HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS=0;HUAWEI_ASCEND_NPU_DUMP_MODEL_FILE_PATH=/work/source/paddle/csc/model/cache/;HUAWEI_ASCEND_NPU_ENABLE_DYNAMIC_SHAPE_RANGE=true;HUAWEI_ASCEND_NPU_PRECISION_MODE=allow_fp32_to_fp16;HUAWEI_ASCEND_NPU_OP_SELECT_IMPL_MODE=high_precision_for_all; [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:67 Context] selected device ids: [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:69 Context] 0 [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:79 Context] profiling path: [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:89 Context] dump model path: /work/source/paddle/csc/model/cache/ [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:99 Context] precision mode: allow_fp32_to_fp16 [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:121 Context] op select impl mode: high_precision_for_all [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:131 Context] op type list for impl mode: [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:141 Context] enable compressw weight: [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:151 Context] auto tune mode: [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:161 Context] enable dynamic shape range: true [I 10/31 23:18: 0.423 ...r/src/driver/huawei_ascend_npu/engine.cc:177 Context] initial buffer length of dynamic shape range: -1 [W 10/31 23:18: 0.423 ...ter/nnadapter/src/runtime/compilation.cc:334 Finish] Warning: Failed to create a program, No model and cache is provided. [W 10/31 23:18: 0.423 ...le-lite/lite/kernels/nnadapter/engine.cc:149 LoadFromCache] Warning: Build model failed(3) ! [I 10/31 23:18: 0.423 ...r/nnadapter/src/operation/elementwise.cc:63 operator()] Cannot broadcast input0: -1, input1: 0 [I 10/31 23:18: 0.423 ...r/nnadapter/src/operation/elementwise.cc:63 operator()] Cannot broadcast input0: -1, input1: 0 [W 10/31 23:18: 0.444 ...nnadapter/nnadapter/src/runtime/model.cc:86 GetSupportedOperations] Warning: Failed to get the supported operations for device 'huawei_ascend_npu', because the HAL interface 'validate_program' is not implemented! [W 10/31 23:18: 0.444 ...kernels/nnadapter/converter/converter.cc:171 Apply] Warning: Failed to get the supported operations for the selected devices, one or more of the selected devices are not supported! [I 10/31 23:18: 0.444 ...r/src/driver/huawei_ascend_npu/driver.cc:70 CreateProgram] Create program for huawei_ascend_npu. [2024-10-31-15:18:08.497.916]1111769 WARNING: Option input_shape_range is deprecated and will be removed in future version,please use input_shape instead [2024-10-31-15:18:08.721.473]1111769 WARNING: Option input_shape_range is deprecated and will be removed in future version,please use input_shape instead [2024-10-31-15:18:08.776.681]1111769 WARNING: Option input_shape_range is deprecated and will be removed in future version,please use input_shape instead [2024-10-31-15:18:08.831.448]1111769 WARNING: Option input_shape_range is deprecated and will be removed in future version,please use input_shape instead [F 10/31 23:18: 8.721 .../src/driver/huawei_ascend_npu/utility.cc:315 BuildOMModelToBuffer] Check failed: (reinterpret_cast(aclgrphBuildModel(ir_graph, options, om_buffer)) == ge::GRAPH_SUCCESS): 1343266818!==0 1343266818 Unknown ATC error code(1343266818) [F 10/31 23:18: 8.721 .../src/driver/huawei_ascend_npu/utility.cc:315 BuildOMModelToBuffer] Check failed: (reinterpret_cast(aclgrphBuildModel(ir_graph, options, om_buffer)) == ge::GRAPH_SUCCESS): 1343266818!==0 1343266818 Unknown ATC error code(1343266818)

[F 10/31 23:18: 8.891 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found. [F 10/31 23:18: 8.891 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

terminate called after throwing an instance of 'nnadapter::logging::Exception' what(): NNAdapter C++ Exception: [F 10/31 23:18: 8.891 ...ter/nnadapter/src/runtime/compilation.cc:98 ~Program] Check failed: device_context: No device found.

Aborted (core dumped)

Process ForkServerPoolWorker-2: Process ForkServerPoolWorker-5: Process ForkServerPoolWorker-4: Process ForkServerPoolWorker-3: Process ForkServerPoolWorker-9: Process ForkServerPoolWorker-7: Traceback (most recent call last): Process ForkServerPoolWorker-8: File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) Process ForkServerPoolWorker-6: File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run()

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) Traceback (most recent call last): BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) Traceback (most recent call last): BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) Traceback (most recent call last): Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, **self._kwargs) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf)

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf)

During handling of the above exception, another exception occurred:

BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last):

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe /root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 41 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

cmcamdy commented 2 days ago

可以试试文档中的动态shape设置方法