search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.23k stars 2.87k forks source link

[Build] Build python interface for Onnxruntime-qnn on aarch64 Linux #21203

Open Misaka17032 opened 3 months ago

Misaka17032 commented 3 months ago

Describe the issue

So I'm using a development kit based on QCS 8550 SoC with an Ubuntu 22.04 on it, and trying to build onnxruntime-qnn on it for python interface.

I'm following the instruction here but was not able to set up the QNN SDK properly. It seems like Qualcomm has not officially supported QNN SDK for aarch64 linux, and thus we are not able to build ort locally on the machine with that instruction.

However, this system did contains the library files like libQnnDsp.so/libQnnHtp.so somewhere, and I can use them for inferencing by tools like qnn-net-run.

Is there any way that I can build the onnxruntime with qnn backend by cross-building from x86 machine or local build from source?

Urgency

The time line for us is pretty tight and we are looking forwards to a solution ASAP.

Target platform

Customized aarch64 Linux with Qualcomm SoC

Build script

build.bat --arm64 --use_qnn --qnn_home=C:\Qualcomm\AIStack\QNN\2.16.4.231110 --build_wheel --cmake_generator "Visual Studio 17 2022" --skip_submodule_sync --config Release --build_dir \build\Windows

Error / output

2024-06-28 16:17:56,155 build [INFO] - Generating CMake build tree
2024-06-28 16:17:56,155 build [INFO] - C:\Users\user\AppData\Local\Programs\Python\Python312\Scripts\cmake.EXE C:\Users\user\Desktop\onnxruntime\cmake -Donnxruntime_RUN_ONNX_TESTS=OFF -Donnxruntime_GENERATE_TEST_REPORTS=ON -DPython_EXECUTABLE=C:\Users\user\AppData\Local\Programs\Python\Python312\python.exe -DPYTHON_EXECUTABLE=C:\Users\user\AppData\Local\Programs\Python\Python312\python.exe -Donnxruntime_USE_MIMALLOC=OFF -Donnxruntime_ENABLE_PYTHON=ON -Donnxruntime_BUILD_CSHARP=OFF -Donnxruntime_BUILD_JAVA=OFF -Donnxruntime_BUILD_NODEJS=OFF -Donnxruntime_BUILD_OBJC=OFF -Donnxruntime_BUILD_SHARED_LIB=OFF -Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF -Donnxruntime_USE_DNNL=OFF -Donnxruntime_USE_NNAPI_BUILTIN=OFF -Donnxruntime_USE_RKNPU=OFF -Donnxruntime_USE_LLVM=OFF -Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF -Donnxruntime_USE_VITISAI=OFF -Donnxruntime_USE_TENSORRT=OFF -Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON -Donnxruntime_USE_TVM=OFF -Donnxruntime_TVM_CUDA_RUNTIME=OFF -Donnxruntime_TVM_USE_HASH=OFF -Donnxruntime_USE_MIGRAPHX=OFF -Donnxruntime_DISABLE_CONTRIB_OPS=OFF -Donnxruntime_DISABLE_ML_OPS=OFF -Donnxruntime_DISABLE_RTTI=OFF -Donnxruntime_DISABLE_EXCEPTIONS=OFF -Donnxruntime_MINIMAL_BUILD=OFF -Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF -Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF -Donnxruntime_REDUCED_OPS_BUILD=OFF -Donnxruntime_USE_DML=OFF -Donnxruntime_USE_WINML=OFF -Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF -Donnxruntime_USE_TELEMETRY=OFF -Donnxruntime_ENABLE_LTO=OFF -Donnxruntime_USE_ACL=OFF -Donnxruntime_USE_ACL_1902=OFF -Donnxruntime_USE_ACL_1905=OFF -Donnxruntime_USE_ACL_1908=OFF -Donnxruntime_USE_ACL_2002=OFF -Donnxruntime_USE_ACL_2308=OFF -Donnxruntime_USE_ARMNN=OFF -Donnxruntime_ARMNN_RELU_USE_CPU=ON -Donnxruntime_ARMNN_BN_USE_CPU=ON -Donnxruntime_USE_JSEP=OFF -Donnxruntime_ENABLE_NVTX_PROFILE=OFF -Donnxruntime_ENABLE_TRAINING=OFF -Donnxruntime_ENABLE_TRAINING_OPS=OFF -Donnxruntime_ENABLE_TRAINING_APIS=OFF -Donnxruntime_ENABLE_CPU_FP16_OPS=OFF -Donnxruntime_USE_NCCL=OFF -Donnxruntime_BUILD_BENCHMARKS=OFF -Donnxruntime_USE_ROCM=OFF -Donnxruntime_GCOV_COVERAGE=OFF -Donnxruntime_USE_MPI=OFF -Donnxruntime_ENABLE_MEMORY_PROFILE=OFF -Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF -Donnxruntime_USE_CUDA_NHWC_OPS=OFF -Donnxruntime_BUILD_WEBASSEMBLY_STATIC_LIB=OFF -Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=ON -Donnxruntime_ENABLE_WEBASSEMBLY_API_EXCEPTION_CATCHING=OFF -Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_THROWING=ON -Donnxruntime_WEBASSEMBLY_RUN_TESTS_IN_BROWSER=OFF -Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=OFF -Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=OFF -Donnxruntime_ENABLE_WEBASSEMBLY_PROFILING=OFF -Donnxruntime_ENABLE_LAZY_TENSOR=OFF -Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF -Donnxruntime_ENABLE_CUDA_PROFILING=OFF -Donnxruntime_ENABLE_ROCM_PROFILING=OFF -Donnxruntime_USE_XNNPACK=OFF -Donnxruntime_USE_WEBNN=OFF -Donnxruntime_USE_CANN=OFF -Donnxruntime_USE_TRITON_KERNEL=OFF -Donnxruntime_DISABLE_FLOAT8_TYPES=OFF -Donnxruntime_DISABLE_SPARSE_TENSORS=OFF -Donnxruntime_DISABLE_OPTIONAL_TYPE=OFF -Donnxruntime_QNN_HOME=C:\Qualcomm\AIStack\QNN\2.16.4.231110 -Donnxruntime_USE_QNN=ON -DCMAKE_VS_GLOBALS=UseMultiToolTask=true;EnforceProcessCountAcrossBuilds=true -Donnxruntime_CROSS_COMPILING=ON -DCMAKE_TLS_VERIFY=ON -DFETCHCONTENT_QUIET=OFF -DONNX_USE_MSVC_STATIC_RUNTIME=OFF -Dprotobuf_MSVC_STATIC_RUNTIME=OFF -Dgtest_force_shared_crt=ON -DCMAKE_MSVC_DEBUG_INFORMATION_FORMAT=ProgramDatabase -A ARM64 -G Visual Studio 17 2022 -DCMAKE_BUILD_TYPE=Release
CMake Warning:
  Ignoring empty string ("") provided on the command line.

CMake Deprecation Warning at CMakeLists.txt:14 (cmake_policy):
  The OLD behavior for policy CMP0104 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.19045.
CMake Error at CMakeLists.txt:20 (project):
  Failed to run MSBuild command:

    C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuild.exe

  to get the value of VCTargetsPath:

    閫傜敤浜?.NET Framework MSBuild 鐗堟湰 17.10.4+10fbfbf2e
    鐢熸垚鍚姩鏃堕棿涓?2024/6/28 16:17:57銆?

    鑺傜偣 1 涓婄殑椤圭洰鈥淐:\build\Windows\Release\CMakeFiles\3.29.6\VCTargetsPath.vcxproj鈥?榛樿鐩爣)銆?
    C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\amd64\Microsoft.Common.CurrentVersion.targets(839,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='ARM64'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\build\Windows\Release\CMakeFiles\3.29.6\VCTargetsPath.vcxproj]
    宸插畬鎴愮敓鎴愰」鐩€淐:\build\Windows\Release\CMakeFiles\3.29.6\VCTargetsPath.vcxproj鈥?榛樿鐩爣)鐨勬搷浣?- 澶辫触銆?

    鐢熸垚澶辫触銆?

    鈥淐:\build\Windows\Release\CMakeFiles\3.29.6\VCTargetsPath.vcxproj鈥?榛樿鐩爣) (1) ->
    (_CheckForInvalidConfigurationAndPlatform 鐩爣) ->
      C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\amd64\Microsoft.Common.CurrentVersion.targets(839,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='ARM64'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\build\Windows\Release\CMakeFiles\3.29.6\VCTargetsPath.vcxproj]

        0 涓鍛?
        1 涓敊璇?

    宸茬敤鏃堕棿 00:00:00.10

  Exit code: 1

-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
  File "C:\Users\user\Desktop\onnxruntime\\tools\ci_build\build.py", line 2958, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\Users\user\Desktop\onnxruntime\\tools\ci_build\build.py", line 2815, in main
    generate_build_tree(
  File "C:\Users\user\Desktop\onnxruntime\\tools\ci_build\build.py", line 1672, in generate_build_tree
    run_subprocess(
  File "C:\Users\user\Desktop\onnxruntime\\tools\ci_build\build.py", line 860, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\Desktop\onnxruntime\tools\python\util\run.py", line 49, in run
    completed_process = subprocess.run(
                        ^^^^^^^^^^^^^^^
  File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\Scripts\\cmake.EXE', 'C:\\Users\\user\\Desktop\\onnxruntime\\cmake', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-DPython_EXECUTABLE=C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\python.exe', '-DPYTHON_EXECUTABLE=C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\python.exe', '-Donnxruntime_USE_MIMALLOC=OFF', '-Donnxruntime_ENABLE_PYTHON=ON', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=OFF', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_OBJC=OFF', '-Donnxruntime_BUILD_SHARED_LIB=OFF', '-Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_TENSORRT=OFF', '-Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_TVM_CUDA_RUNTIME=OFF', '-Donnxruntime_TVM_USE_HASH=OFF', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ACL_1902=OFF', '-Donnxruntime_USE_ACL_1905=OFF', '-Donnxruntime_USE_ACL_1908=OFF', '-Donnxruntime_USE_ACL_2002=OFF', '-Donnxruntime_USE_ACL_2308=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_USE_JSEP=OFF', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_TRAINING_APIS=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=OFF', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-Donnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=OFF', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_USE_CUDA_NHWC_OPS=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY_STATIC_LIB=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=ON', '-Donnxruntime_ENABLE_WEBASSEMBLY_API_EXCEPTION_CATCHING=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_THROWING=ON', '-Donnxruntime_WEBASSEMBLY_RUN_TESTS_IN_BROWSER=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_PROFILING=OFF', '-Donnxruntime_ENABLE_LAZY_TENSOR=OFF', '-Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF', '-Donnxruntime_ENABLE_CUDA_PROFILING=OFF', '-Donnxruntime_ENABLE_ROCM_PROFILING=OFF', '-Donnxruntime_USE_XNNPACK=OFF', '-Donnxruntime_USE_WEBNN=OFF', '-Donnxruntime_USE_CANN=OFF', '-Donnxruntime_USE_TRITON_KERNEL=OFF', '-Donnxruntime_DISABLE_FLOAT8_TYPES=OFF', '-Donnxruntime_DISABLE_SPARSE_TENSORS=OFF', '-Donnxruntime_DISABLE_OPTIONAL_TYPE=OFF', '-Donnxruntime_QNN_HOME=C:\\Qualcomm\\AIStack\\QNN\\2.16.4.231110', '-Donnxruntime_USE_QNN=ON', '-DCMAKE_VS_GLOBALS=UseMultiToolTask=true;EnforceProcessCountAcrossBuilds=true', '-Donnxruntime_CROSS_COMPILING=ON', '-DCMAKE_TLS_VERIFY=ON', '-DFETCHCONTENT_QUIET=OFF', '-DONNX_USE_MSVC_STATIC_RUNTIME=OFF', '-Dprotobuf_MSVC_STATIC_RUNTIME=OFF', '-Dgtest_force_shared_crt=ON', '-DCMAKE_MSVC_DEBUG_INFORMATION_FORMAT=ProgramDatabase', '-A', 'ARM64', '-G', 'Visual Studio 17 2022', '-DCMAKE_BUILD_TYPE=Release', '']' returned non-zero exit status 1.

Visual Studio Version

No response

GCC / Compiler Version

No response

jywu-msft commented 3 months ago

the QNN docs don't mention aarch64 Linux support, but you're right, the QNN SDK does seem to have some aarch64--linux directories under lib cross compiling on Windows isn't going to work in your case if you are targeting a Linux system. maybe it's possible to modify the CMakeLists.txt https://github.com/microsoft/onnxruntime/blob/7e93cd7f8b21d00b8d406cb043ff5f0e61217d85/cmake/CMakeLists.txt#L832 to target one of the aarch64--linux , currently it assumes non x86_64 Linux is aarch64-android but that's not the case for you.

Misaka17032 commented 3 months ago

Hi, thank you for the advise.

so I did try to do the compiling on a x86_64 ubuntu machine, and edit the CMakeLists.txt as you suggested:

      else()
        if (${CMAKE_SYSTEM_NAME} STREQUAL "Android")
          set(QNN_ARCH_ABI aarch64-android-clang6.0)
        elseif (${CMAKE_SYSTEM_NAME} STREQUAL "Linux")
          if (${GEN_PLATFORM} STREQUAL "x86_64")
            set(QNN_ARCH_ABI aarch64-linux-clang)  # edited
          else()
            set(QNN_ARCH_ABI aarch64-linux-clang)  # edited
          endif()
        endif()
      endif()
    endif()

and here is the command I use:

bash build.sh --arm64 --use_qnn --qnn_home=/opt/qcom/aistack/qnn/2.16.4.231110 --parallel --build_wheel --skip_submodule_sync --config Release

The building process is all passed until the step Linking CXX shared module onnxruntime_pybind11_state.so:

[100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/mnt/user/onnxruntime-qnn/onnxruntime/onnxruntime/test/flatbuffers/flatbuffer_utils_test.cc.o
[100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/mnt/user/onnxruntime-qnn/onnxruntime/onnxruntime/test/unittest_main/test_main.cc.o
[100%] Linking CXX shared module onnxruntime_pybind11_state.so
CMake Error: cmake version 3.29.6
Usage: /mnt/user/anaconda3/envs/onnxqnn/lib/python3.10/site-packages/cmake/data/bin/cmake -E <command> [arguments...]
Available commands: 
  # just the usage of cmake, omited

make[2]: *** [CMakeFiles/onnxruntime_pybind11_state.dir/build.make:407: onnxruntime_pybind11_state.so] Error 1
make[2]: *** Deleting file 'onnxruntime_pybind11_state.so'
make[1]: *** [CMakeFiles/Makefile2:2208: CMakeFiles/onnxruntime_pybind11_state.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[100%] Linking CXX executable onnxruntime_test_all
CMake Error: cmake version 3.29.6
Usage: /mnt/user/anaconda3/envs/onnxqnn/lib/python3.10/site-packages/cmake/data/bin/cmake -E <command> [arguments...]
Available commands: 
  # just the usage of cmake, omited

make[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:5660: onnxruntime_test_all] Error 1
make[2]: *** Deleting file 'onnxruntime_test_all'
make[1]: *** [CMakeFiles/Makefile2:2363: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
Weisonorz commented 3 months ago

Do you have any update on solving this issue? (compiling onnx_qnn on aarch linux) Our team is having the same problem with setting up the cmake file. By the way, does this onnx_qnn allow dynamic model input?

jywu-msft commented 3 months ago

are you all trying to cross compile or build natively on aarch64 Linux? remove --arm64 from build option and try to build natively?

Misaka17032 commented 2 months ago

For me, I tried to build natively on aarch64 Linux first but then realized that I can't setup the QNN SDK on that environment.

jywu-msft commented 2 months ago

For me, I tried to build natively on aarch64 Linux first but then realized that I can't setup the QNN SDK on that environment.

what setup are you missing. i think you can try copying the qnn sdk installation directory (e.g. /path/to/qnn/2.23.0.240531) from an installation to different platform (e.g. Linux x64) to the aarch64 linux environment , make the cmake modification mentioned above and give it another try.

Misaka17032 commented 2 months ago

yes, I did try that before and it seems to failed. Thank you for the advice though, I'll still try to work on that.

jywu-msft commented 2 months ago

yes, I did try that before and it seems to failed. Thank you for the advice though, I'll still try to work on that.

what kind of failure do you get. we can reach out to qualcomm and ask for guidance.

Misaka17032 commented 2 months ago

[edited]

So I did some debugging and resolve the previous error, and the failure now is listed below:

/home/user/ort_build/onnxruntime/onnxruntime/core/providers/qnn/builder/qnn_utils.cc: In function ‘std::pair<float, float> onnxruntime::qnn::utils::CheckMinMax(float, float)’:
/home/user/ort_build/onnxruntime/onnxruntime/core/providers/qnn/builder/qnn_utils.cc:466:59: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
  466 | std::pair<float, float> CheckMinMax(float rmin, float rmax) {
      |                                                           ^
/home/user/ort_build/onnxruntime/onnxruntime/core/providers/qnn/qnn_execution_provider.cc: In function ‘void onnxruntime::ParseHtpArchitecture(const string&, QnnHtpDevice_Arch_t&)’:
/home/user/ort_build/onnxruntime/onnxruntime/core/providers/qnn/qnn_execution_provider.cc:158:20: error: ‘QNN_HTP_DEVICE_ARCH_V75’ was not declared in this scope; did you mean ‘QNN_HTP_DEVICE_ARCH_V73’?
  158 |     qnn_htp_arch = QNN_HTP_DEVICE_ARCH_V75;
      |                    ^~~~~~~~~~~~~~~~~~~~~~~
      |                    QNN_HTP_DEVICE_ARCH_V73
gmake[2]: *** [CMakeFiles/onnxruntime_providers_qnn.dir/build.make:552: CMakeFiles/onnxruntime_providers_qnn.dir/home/user/ort_build/onnxruntime/onnxruntime/core/providers/qnn/qnn_execution_provider.cc.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2052: CMakeFiles/onnxruntime_providers_qnn.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
Misaka17032 commented 2 months ago

Okay, so I've been trying to figure out the dependencies and configurations, and finally I did build a python wheel with the following command:

bash build.sh --build --arm64 --use_qnn --qnn_home=/path/to/qnn/2.22.6.240515/ --parallel --build_wheel --skip_submodule_sync --config Release

However, when I tried to follow the instruction and run a basic mobilenet_v3_small, the following error raised:

python I  8330  8330 BufferAllocator.cpp:100] Using DMA-BUF heap named: system
/prj/qct/webtech_hyd18/mlg_user_admin/qaisw_source_repo/qaisw_repo_point_release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.4.0/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
2024-07-05 17:52:03.839963468 [E:onnxruntime:, qnn_execution_provider.cc:597 GetCapability] QNN SetupBackend failed Failed to create device. Error: 14001

The script I run is:

import onnxruntime
import numpy as np

options = onnxruntime.SessionOptions()

# (Optional) Enable configuration that raises an exception if the model can't be
# run entirely on the QNN HTP backend.
# options.add_session_config_entry("session.disable_cpu_ep_fallback", "1")

# Create an ONNX Runtime session.
session = onnxruntime.InferenceSession("mobnet.qdq.onnx",
                                       sess_options=options,
                                       providers=["QNNExecutionProvider"],
                                       provider_options=[{"backend_path": "./lib/libQnnHtp.so"}])

# Run the model with your input.
# TODO: Use numpy to load your actual input from a file or generate random input.
input0 = np.ones((1,3,224,224), dtype=np.float32)
while True:
    result = session.run(None, {"input.1": input0})

# Print output.
print(result)
Lucashien commented 2 months ago

Did you solve the issue with "QNN SetupBackend failed Failed to create device"? I'm facing the same problem on Windows AArch64 (Snapdragon 8cx Gen3). Thanks!

skyline75489 commented 2 months ago

The error code 14001 in QNN means QNN_DEVICE_ERROR_INVALID_CONFIG: Invalid config values. But this isn't really necessarily caused by configs. Most likely you don't have the ADSP_LIBRARY_PATH env var correctly, and QNN can't load all the necessary libraries it needs.

On Andriod, you'll probably also need to reference libcdsprpc.so in AndroidManifest.xml:

<uses-library
    android:name="libcdsprpc.so"
    android:required="true"/>

See: https://developer.qualcomm.com/sites/default/files/docs/snpe/dsp_runtime.html