search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.43k stars 2.9k forks source link

[Build] Unable to cross compile for AArch64 Linux #16308

Closed cyrusbehr closed 11 months ago

cyrusbehr commented 1 year ago

Describe the issue

I am following the instructions here on how to compile onnxruntime v1.15.0 from host x86_64 linux for target aarch64 linux, but the build is failing. I imagine this build is part of your CI pipeline, so probably just the documentation is out of date.

Urgency

medium - high We use onnxruntime for a commercial product and we have deadlines.

Target platform

Host: x86_64 linux (Ubuntu 20.04), Target: aarch64-linux-gnu

Build script

Here are the steps to reproduce:

  1. Clone the github repo and submodules, checkout v1.15.0.

  2. Ensure we have the cross compiler installed. Output from running this command: aarch64-linux-gnu-gcc -v is:

    Using built-in specs.
COLLECT_GCC=aarch64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/aarch64-linux-gnu/9/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --without-target-system-zlib --enable-libpth-m2 --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=aarch64-linux-gnu --program-prefix=aarch64-linux-gnu- --includedir=/usr/aarch64-linux-gnu/include
Thread model: posix
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

  3. Install prebuild protobuf library. Docs say to look at this file / dir to determine version, but it no longer exists in the repo: image

image

Looking through the repo ^ looks like you updated it to version 3.21, so let's download that version

Download and extract: https://github.com/protocolbuffers/protobuf/releases/download/v21.12/protoc-21.12-linux-x86_64.zip

  1. Define our cmake toolchain file as follows: 
    
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
SET(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
SET(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

5. Create a build directory, run the following cmake command:

cmake -Donnxruntime_GCC_STATIC_CPP_RUNTIME=ON -DCMAKE_BUILD_TYPE=Release -Dprotobuf_WITH_ZLIB=OFF -DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake -Donnxruntime_ENABLE_PYTHON=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_DEV_MODE=OFF -DONNX_CUSTOM_PROTOC_EXECUTABLE=/home/cyrus/work/libs/protoc-21.12-linux-x86_64/bin/protoc ../cmake/


Output:

CMake Deprecation Warning at CMakeLists.txt:21 (cmake_policy): The OLD behavior for policy CMP0104 will be removed from a future version of CMake.

The cmake-policies(7) manual explains that the OLD behaviors of all policies are deprecated and that a policy should be set to OLD only under specific short-term circumstances. Projects should be ported to the NEW behavior and not rely on setting a policy to OLD.

-- The C compiler identification is GNU 9.4.0 -- The CXX compiler identification is GNU 9.4.0 -- The ASM compiler identification is GNU -- Found assembler: /usr/bin/aarch64-linux-gnu-gcc -- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done Building ONNX Runtime for -- Performing Test COMPILER_SUPPORT_MF16C -- Performing Test COMPILER_SUPPORT_MF16C - Failed F16C instruction set is not supported. -- Performing Test COMPILER_SUPPORT_FMA -- Performing Test COMPILER_SUPPORT_FMA - Failed FMA instruction set is not supported. -- Performing Test COMPILER_SUPPORT_AVX -- Performing Test COMPILER_SUPPORT_AVX - Failed AVX instruction set is not supported. One or more AVX/F16C instruction flags are not supported. -- Performing Test onnxruntime_HAVE_BUILTIN_ATOMICS -- Performing Test onnxruntime_HAVE_BUILTIN_ATOMICS - Success -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE
-- Performing Test Iconv_IS_BUILT_IN -- Performing Test Iconv_IS_BUILT_IN - Success -- Found Iconv: built in to C library
-- Found Patch: /usr/bin/patch
Patch found: /usr/bin/patch Doing crosscompiling -- Found Python: /usr/bin/python3.8 (found suitable version "3.8.10", minimum required is "3.8") found components: Interpreter Loading Dependencies URLs ... Loading Dependencies ... CMake Warning at /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src/absl/copts/AbseilConfigureCopts.cmake:70 (message): Value of CMAKE_SYSTEM_PROCESSOR () is unknown and cannot be used to set ABSL_RANDOM_RANDEN_COPTS Call Stack (most recent call first): /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src/CMake/AbseilHelpers.cmake:18 (include) /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src/CMakeLists.txt:81 (include)

-- Abseil source dir:/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src

date: USE_SYSTEM_TZ_DB ON

date: USE_TZ_DB_IN_DOT OFF

date: BUILD_SHARED_LIBS OFF

date: ENABLE_DATE_TESTING OFF

-- -- 3.21.12.0 -- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT -- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT - Success -- Performing Test protobuf_HAVE_BUILTIN_ATOMICS -- Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success -- Using the single-header code from /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/nlohmann_json-src/single_include/ -- Check if compiler accepts -pthread -- Check if compiler accepts -pthread - yes -- Looking for strtof_l -- Looking for strtof_l - found -- Looking for strtoull_l -- Looking for strtoull_l - found -- Using toolchain file: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/aarch64.cmake. -- Found Python: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter CMake Warning at /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/pytorch_cpuinfo-src/CMakeLists.txt:59 (MESSAGE): Target processor architecture is not specified. cpuinfo will compile, but cpuinfo_initialize() will always fail.

-- Found PythonInterp: /usr/bin/python (found version "3.8.10") -- Using custom protoc executable Generated: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/onnx-build/onnx/onnx-ml.proto Generated: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/onnx-build/onnx/onnx-operators-ml.proto Generated: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/onnx-build/onnx/onnx-data.proto -- -- **** Summary **** -- CMake version : 3.26.3 -- CMake command : /home/cyrus/.local/lib/python3.8/site-packages/cmake/data/bin/cmake -- System : Linux -- C++ compiler : /usr/bin/aarch64-linux-gnu-g++ -- C++ compiler version : 9.4.0 -- CXX flags : -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -Wnon-virtual-dtor -- Build type : Release -- Compile definitions : ORT_ENABLE_STREAM;ORT_NO_RTTI;EIGEN_MPL2_ONLY;_GNU_SOURCE;__STDC_FORMAT_MACROS -- CMAKE_PREFIX_PATH : -- CMAKE_INSTALL_PREFIX : /usr/local -- CMAKE_MODULE_PATH : /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/cmake/external -- -- ONNX version : 1.14.0 -- ONNX NAMESPACE : onnx -- ONNX_USE_LITE_PROTO : ON -- USE_PROTOBUF_SHARED_LIBS : OFF -- Protobuf_USE_STATIC_LIBS : ON -- ONNX_DISABLE_EXCEPTIONS : OFF -- ONNX_WERROR : OFF -- ONNX_BUILD_TESTS : OFF -- ONNX_BUILD_BENCHMARKS : OFF -- -- Protobuf compiler : -- Protobuf includes : -- Protobuf libraries : -- BUILD_ONNX_PYTHON : OFF Finished fetching external dependencies -- Performing Test HAS_UNUSED_BUT_SET_PARAMETER -- Performing Test HAS_UNUSED_BUT_SET_PARAMETER - Success -- Performing Test HAS_UNUSED_BUT_SET_VARIABLE -- Performing Test HAS_UNUSED_BUT_SET_VARIABLE - Success -- Performing Test HAS_UNUSED_VARIABLE -- Performing Test HAS_UNUSED_VARIABLE - Success -- Performing Test HAS_CAST_FUNCTION_TYPE -- Performing Test HAS_CAST_FUNCTION_TYPE - Success -- Performing Test HAS_PARENTHESES -- Performing Test HAS_PARENTHESES - Success -- Performing Test HAS_USELESS_CAST -- Performing Test HAS_USELESS_CAST - Success -- Performing Test HAS_NONNULL_COMPARE -- Performing Test HAS_NONNULL_COMPARE - Success -- Performing Test HAS_TAUTOLOGICAL_POINTER_COMPARE -- Performing Test HAS_TAUTOLOGICAL_POINTER_COMPARE - Failed -- Performing Test HAS_CATCH_VALUE -- Performing Test HAS_CATCH_VALUE - Success -- Performing Test HAS_MISSING_BRACES -- Performing Test HAS_MISSING_BRACES - Success -- Performing Test HAS_IGNORED_ATTRIBUTES -- Performing Test HAS_IGNORED_ATTRIBUTES - Success -- Performing Test HAS_DEPRECATED_COPY -- Performing Test HAS_DEPRECATED_COPY - Success -- Performing Test HAS_DEPRECATED_DECLARATIONS -- Performing Test HAS_DEPRECATED_DECLARATIONS - Success -- Performing Test HAS_CLASS_MEMACCESS -- Performing Test HAS_CLASS_MEMACCESS - Success -- Performing Test HAS_MAYBE_UNINITIALIZED -- Performing Test HAS_MAYBE_UNINITIALIZED - Success -- Performing Test HAS_STRICT_ALIASING -- Performing Test HAS_STRICT_ALIASING - Success NVCC_ERROR = NVCC_OUT = No such file or directory -- Performing Test HAS_AMBIGUOUS_REVERSED_OPERATOR -- Performing Test HAS_AMBIGUOUS_REVERSED_OPERATOR - Failed -- Performing Test HAS_DEPRECATED_ANON_ENUM_ENUM_CONVERSION -- Performing Test HAS_DEPRECATED_ANON_ENUM_ENUM_CONVERSION - Failed -- Performing Test HAS_UNDEFINED_VAR_TEMPLATE -- Performing Test HAS_UNDEFINED_VAR_TEMPLATE - Failed -- Performing Test HAS_FORMAT_TRUNCATION -- Performing Test HAS_FORMAT_TRUNCATION - Success -- Performing Test HAS_BITWISE_INSTEAD_OF_LOGICAL -- Performing Test HAS_BITWISE_INSTEAD_OF_LOGICAL - Failed -- Performing Test HAS_ENUM_CONSTEXPR_CONVERSION -- Performing Test HAS_ENUM_CONSTEXPR_CONVERSION - Failed -- Performing Test HAS_DEPRECATED_BUILTINS -- Performing Test HAS_DEPRECATED_BUILTINS - Failed -- Looking for reallocarray -- Looking for reallocarray - found -- Found Git: /usr/bin/git (found version "2.25.1") CMake Warning at CMakeLists.txt:1475 (message): MPI and NCCL disabled on Win build.

-- Looking for clock_gettime in rt -- Looking for clock_gettime in rt - found CMake Warning (dev) at onnxruntime_mlas.cmake:587: Syntax Warning in cmake code at column 107

Argument not separated from preceding token by whitespace. Call Stack (most recent call first): CMakeLists.txt:1609 (include) This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning at onnxruntime_mlas.cmake:63 (message): AMX instructions NOT supported due to lack of compiler tool chain! Call Stack (most recent call first): CMakeLists.txt:1609 (include)

-- Configuring done (22.1s) -- Generating done (0.2s) CMake Warning: Manually-specified variables were not used by the project:

onnxruntime_DEV_MODE
onnxruntime_GCC_STATIC_CPP_RUNTIME

-- Build files have been written to: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp


6. Run `make`, command fails at 70% on this error:

[ 64%] Built target onnxruntime_session [ 64%] Built target onnxruntime_generate_def [ 66%] Built target nsync_cpp [ 69%] Built target re2 [ 69%] Built target clog [ 70%] Built target cpuinfo [ 70%] Building C object CMakeFiles/onnxruntime.dir/generated_source.c.o [ 70%] Linking CXX shared library libonnxruntime.so /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: libonnxruntime_common.a(cpuid_info.cc.o): in function onnxruntime::CPUIDInfo::ArmLinuxInit()': cpuid_info.cc:(.text._ZN11onnxruntime9CPUIDInfo12ArmLinuxInitEv+0x42c): undefined reference tocpuinfo_isa' /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: cpuid_info.cc:(.text._ZN11onnxruntime9CPUIDInfo12ArmLinuxInitEv+0x438): undefined reference to cpuinfo_isa' /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: _deps/pytorch_cpuinfo-build/libcpuinfo.a(init.c.o): in functioncpuinfo_initialize': init.c:(.text.cpuinfo_initialize+0x4): undefined reference to cpuinfo_arm_linux_init' /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: init.c:(.text.cpuinfo_initialize+0x10): undefined reference tocpuinfo_arm_linux_init' /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: libonnxruntime.so.1.15.0: hidden symbol `cpuinfo_arm_linux_init' isn't defined /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: final link failed: bad value collect2: error: ld returned 1 exit status make[2]: [CMakeFiles/onnxruntime.dir/build.make:153: libonnxruntime.so.1.15.0] Error 1 make[1]: [CMakeFiles/Makefile2:2021: CMakeFiles/onnxruntime.dir/all] Error 2 make: *** [Makefile:166: all] Error 2



### Error / output

See above box

### Visual Studio Version

_No response_

### GCC / Compiler Version

gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
cyrusbehr commented 1 year ago

I also tried the fix suggested in this issue, but it didn't solve things.

snnn commented 1 year ago

Could you go to /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp and run make VERBOSE=1 and post the link command of libonnxruntime.so so that I can see if cpuinfo was added to the link command of "libonnxruntime.so" or not?

snnn commented 1 year ago

You may also add --cmake_extra_defines -Donnxruntime_ENABLE_CPUINFO=OFF to your build command

cyrusbehr commented 1 year ago

@snnn here is the output of make VERBOSE=1


[ 70%] Building C object CMakeFiles/onnxruntime.dir/generated_source.c.o
/usr/bin/aarch64-linux-gnu-gcc -DCPUINFO_SUPPORTED_PLATFORM=0 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DFILE_NAME=\"onnxruntime.dll\" -DJSON_DIAGNOSTICS=0 -DJSON_USE_IMPLICIT_CONVERSIONS=1 -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DONNX_USE_LITE_PROTO=1 -DORT_ENABLE_STREAM -DORT_NO_RTTI -DPLATFORM_POSIX -DVER_BUILD=0 -DVER_MAJOR=0 -DVER_MINOR=0 -DVER_PRIVATE=0 -DVER_STRING="\"Internal Build\"" -D_GNU_SOURCE -D__ONNX_NO_DOC_STRINGS -Donnxruntime_EXPORTS -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/include/onnxruntime -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/include/onnxruntime/core/session -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/pytorch_cpuinfo-src/include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/google_nsync-src/public -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64 -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/abseil_cpp-src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/eigen-src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/safeint-src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/microsoft_wil-src/include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/nlohmann_json-src/single_include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/onnx-src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/onnx-build -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/protobuf-src/src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/re2-src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/mp11-src/include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/flatbuffers-src/include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/date-src/include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/pytorch_cpuinfo-src/deps/clog/include -isystem /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/gsl-src/include -ffunction-sections -fdata-sections -O3 -DNDEBUG -std=gnu99 -fPIC -Wall -Wextra -Wno-error=comment -Werror -MD -MT CMakeFiles/onnxruntime.dir/generated_source.c.o -MF CMakeFiles/onnxruntime.dir/generated_source.c.o.d -o CMakeFiles/onnxruntime.dir/generated_source.c.o -c /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/generated_source.c
[ 70%] Linking CXX shared library libonnxruntime.so
/home/cyrus/.local/lib/python3.8/site-packages/cmake/data/bin/cmake -E cmake_link_script CMakeFiles/onnxruntime.dir/link.txt --verbose=1
/usr/bin/aarch64-linux-gnu-g++ -fPIC  -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -O3 -DNDEBUG  -Xlinker --version-script=/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/onnxruntime.lds -Xlinker --no-undefined -Xlinker --gc-sections -z noexecstack  -Wl,-rpath='$ORIGIN' -shared -Wl,-soname,libonnxruntime.so.1.15.0 -o libonnxruntime.so.1.15.0 CMakeFiles/onnxruntime.dir/generated_source.c.o  libonnxruntime_session.a libonnxruntime_optimizer.a libonnxruntime_providers.a libonnxruntime_framework.a libonnxruntime_graph.a libonnxruntime_util.a libonnxruntime_mlas.a libonnxruntime_common.a libonnxruntime_flatbuffers.a _deps/onnx-build/libonnx.a _deps/onnx-build/libonnx_proto.a _deps/protobuf-build/libprotobuf-lite.a _deps/re2-build/libre2.a _deps/flatbuffers-build/libflatbuffers.a _deps/abseil_cpp-build/absl/base/libabsl_base.a _deps/abseil_cpp-build/absl/base/libabsl_throw_delegate.a _deps/abseil_cpp-build/absl/container/libabsl_raw_hash_set.a _deps/abseil_cpp-build/absl/hash/libabsl_hash.a _deps/abseil_cpp-build/absl/hash/libabsl_city.a _deps/abseil_cpp-build/absl/hash/libabsl_low_level_hash.a _deps/abseil_cpp-build/absl/base/libabsl_raw_logging_internal.a _deps/google_nsync-build/libnsync_cpp.a -ldl -lrt _deps/pytorch_cpuinfo-build/libcpuinfo.a _deps/pytorch_cpuinfo-build/deps/clog/libclog.a _deps/abseil_cpp-build/absl/strings/libabsl_cord.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_info.a _deps/abseil_cpp-build/absl/strings/libabsl_cord_internal.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_functions.a _deps/abseil_cpp-build/absl/strings/libabsl_cordz_handle.a _deps/abseil_cpp-build/absl/container/libabsl_raw_hash_set.a _deps/abseil_cpp-build/absl/container/libabsl_hashtablez_sampler.a _deps/abseil_cpp-build/absl/profiling/libabsl_exponential_biased.a _deps/abseil_cpp-build/absl/synchronization/libabsl_synchronization.a _deps/abseil_cpp-build/absl/synchronization/libabsl_graphcycles_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_stacktrace.a _deps/abseil_cpp-build/absl/debugging/libabsl_symbolize.a _deps/abseil_cpp-build/absl/base/libabsl_malloc_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_debugging_internal.a _deps/abseil_cpp-build/absl/debugging/libabsl_demangle_internal.a _deps/abseil_cpp-build/absl/time/libabsl_time.a _deps/abseil_cpp-build/absl/time/libabsl_civil_time.a _deps/abseil_cpp-build/absl/time/libabsl_time_zone.a _deps/abseil_cpp-build/absl/hash/libabsl_hash.a _deps/abseil_cpp-build/absl/types/libabsl_bad_optional_access.a _deps/abseil_cpp-build/absl/strings/libabsl_strings.a _deps/abseil_cpp-build/absl/base/libabsl_throw_delegate.a _deps/abseil_cpp-build/absl/strings/libabsl_strings_internal.a _deps/abseil_cpp-build/absl/types/libabsl_bad_variant_access.a _deps/abseil_cpp-build/absl/hash/libabsl_city.a _deps/abseil_cpp-build/absl/hash/libabsl_low_level_hash.a _deps/abseil_cpp-build/absl/base/libabsl_base.a _deps/abseil_cpp-build/absl/base/libabsl_spinlock_wait.a _deps/abseil_cpp-build/absl/numeric/libabsl_int128.a _deps/abseil_cpp-build/absl/base/libabsl_raw_logging_internal.a _deps/abseil_cpp-build/absl/base/libabsl_log_severity.a -lpthread 
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: libonnxruntime_common.a(cpuid_info.cc.o): in function `onnxruntime::CPUIDInfo::ArmLinuxInit()':
cpuid_info.cc:(.text._ZN11onnxruntime9CPUIDInfo12ArmLinuxInitEv+0x42c): undefined reference to `cpuinfo_isa'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: cpuid_info.cc:(.text._ZN11onnxruntime9CPUIDInfo12ArmLinuxInitEv+0x438): undefined reference to `cpuinfo_isa'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: _deps/pytorch_cpuinfo-build/libcpuinfo.a(init.c.o): in function `cpuinfo_initialize':
init.c:(.text.cpuinfo_initialize+0x4): undefined reference to `cpuinfo_arm_linux_init'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: init.c:(.text.cpuinfo_initialize+0x10): undefined reference to `cpuinfo_arm_linux_init'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: libonnxruntime.so.1.15.0: hidden symbol `cpuinfo_arm_linux_init' isn't defined
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/onnxruntime.dir/build.make:153: libonnxruntime.so.1.15.0] Error 1
make[2]: Leaving directory '/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64'
make[1]: *** [CMakeFiles/Makefile2:2021: CMakeFiles/onnxruntime.dir/all] Error 2
make[1]: Leaving directory '/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64'
make: *** [Makefile:166: all] Error 2
cyrusbehr commented 1 year ago

Here is the output of make VERBOSE=1 when -Donnxruntime_ENABLE_CPUINFO=OFF is set as well:


[ 15%] Building CXX object CMakeFiles/onnxruntime_common.dir/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc.o
/usr/bin/aarch64-linux-gnu-g++ -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DNSYNC_ATOMIC_CPP11 -DORT_ENABLE_STREAM -DORT_NO_RTTI -DPLATFORM_POSIX -D_GNU_SOURCE -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/include/onnxruntime -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/include/onnxruntime/core/session -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/google_nsync-src/public -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64 -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/abseil_cpp-src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/date-src/include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/microsoft_wil-src/include -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/eigen-src -I/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/safeint-src -isystem /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64/_deps/gsl-src/include -ffunction-sections -fdata-sections -O3 -DNDEBUG -std=gnu++17 -fPIC -fno-rtti -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -pthread -Werror -MD -MT CMakeFiles/onnxruntime_common.dir/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc.o -MF CMakeFiles/onnxruntime_common.dir/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc.o.d -o CMakeFiles/onnxruntime_common.dir/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc.o -c /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc: In member function ‘void onnxruntime::CPUIDInfo::ArmLinuxInit()’:
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:143:18: error: ‘cpuinfo_get_uarchs_count’ was not declared in this scope
  143 |     is_hybrid_ = cpuinfo_get_uarchs_count() > 1;
      |                  ^~~~~~~~~~~~~~~~~~~~~~~~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:144:25: error: ‘cpuinfo_has_arm_neon_dot’ was not declared in this scope
  144 |     has_arm_neon_dot_ = cpuinfo_has_arm_neon_dot();
      |                         ^~~~~~~~~~~~~~~~~~~~~~~~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:145:17: error: ‘cpuinfo_has_arm_neon_fp16_arith’ was not declared in this scope
  145 |     has_fp16_ = cpuinfo_has_arm_neon_fp16_arith();
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:146:31: error: ‘cpuinfo_get_cores_count’ was not declared in this scope
  146 |     const uint32_t core_cnt = cpuinfo_get_cores_count();
      |                               ^~~~~~~~~~~~~~~~~~~~~~~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:150:46: error: ‘cpuinfo_get_processor’ was not declared in this scope; did you mean ‘cpuinfo_processor’?
  150 |       const struct cpuinfo_processor* proc = cpuinfo_get_processor(c);
      |                                              ^~~~~~~~~~~~~~~~~~~~~
      |                                              cpuinfo_processor
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:154:46: error: invalid use of incomplete type ‘const struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_processor’
  154 |       const struct cpuinfo_core* corep = proc->core;
      |                                              ^~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:150:20: note: forward declaration of ‘struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_processor’
  150 |       const struct cpuinfo_processor* proc = cpuinfo_get_processor(c);
      |                    ^~~~~~~~~~~~~~~~~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:158:25: error: invalid use of incomplete type ‘const struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_processor’
  158 |       auto coreid = proc->linux_id;
      |                         ^~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:150:20: note: forward declaration of ‘struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_processor’
  150 |       const struct cpuinfo_processor* proc = cpuinfo_get_processor(c);
      |                    ^~~~~~~~~~~~~~~~~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:159:25: error: invalid use of incomplete type ‘const struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_core’
  159 |       auto uarch = corep->uarch;
      |                         ^~
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc:154:20: note: forward declaration of ‘struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_core’
  154 |       const struct cpuinfo_core* corep = proc->core;
      |                    ^~~~~~~~~~~~
make[2]: *** [CMakeFiles/onnxruntime_common.dir/build.make:76: CMakeFiles/onnxruntime_common.dir/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/onnxruntime/core/common/cpuid_info.cc.o] Error 1
make[2]: Leaving directory '/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64'
make[1]: *** [CMakeFiles/Makefile2:1671: CMakeFiles/onnxruntime_common.dir/all] Error 2
make[1]: Leaving directory '/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_aarch64'
make: *** [Makefile:166: all] Error 2
snnn commented 1 year ago

This is strange, libcpuinfo.a is in the link command line. But the symbols were still not found. I didn't write this code. I will try to understand it and reach back to you later.

cyrusbehr commented 1 year ago

@snnn did you have a chance to look into this? We need to be able to cross compile the library as part of a commercial product, and deadlines are fast approaching.

snnn commented 1 year ago

Sorry I wasn't able to figure why it didn't work. I suggest disabling it if it doesn't work for you.

cyrusbehr commented 1 year ago

How can I disable it?

cyrusbehr commented 1 year ago

You mean disabling our entire CI build? Or disabling the specific option which is causing the onnxruntime build to fail?

hfwanguanghui commented 1 year ago

You mean disabling our entire CI build? Or disabling the specific option which is causing the onnxruntime build to fail?

Hi，I had also mat this problem。 did you had resolved it？

cyrusbehr commented 1 year ago

@hfwanguanghui No it has not been resolved. I ended up using the prebuilt aarch64 binary. They unfortunately don't ship binaries for aarch32 so do I still require this issue resolved eventually.

sammylievens commented 1 year ago

I was able to cross-compile the ONNX runtime 1.15.1 for ARM 32-bit, but had to make sure no more recent version of Protobuf was installed locally on my PC. For cross-compiling the ONNX runtime 1.11.0 this was however not needed. I think they messed up a bit their CMake build environment, they made some changes in that external deps folder.

I used the following commands (sorry for the hard-coded paths)

wget https://github.com/protocolbuffers/protobuf/releases/download/v21.12/protoc-21.12-linux-x86_64.zip
unzip protoc-21.12-linux-x86_64.zip -d protoc-3.21.12
rm -rf protoc-21.12-linux-x86_64.zip

git clone --recursive https://github.com/Microsoft/onnxruntime -b v1.15.1 onnxruntime-v1.15.1
cd onnxruntime-v1.15.1
mkdir build
cd build
ARMCC_FLAGS="-mcpu=cortex-a53 -mfpu=crypto-neon-fp-armv8 -fPIC"
ARMCC_PREFIX=/opt/OSELAS.Toolchain-2018.12.0/arm-v7a-linux-gnueabihf/gcc-8.3.0-glibc-2.28-binutils-2.31.1-kernel-4.19-sanitized-cortex-a53/bin/arm-v7a-linux-gnueabihf-
cmake -DCMAKE_C_COMPILER=${ARMCC_PREFIX}gcc \
  -DCMAKE_CXX_COMPILER=${ARMCC_PREFIX}g++ \
  -DCMAKE_C_FLAGS="${ARMCC_FLAGS}" \
  -DCMAKE_CXX_FLAGS="${ARMCC_FLAGS}" \
  -DCMAKE_SYSTEM_NAME=Linux \
  -DCMAKE_SYSTEM_PROCESSOR=armv7 \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
  -Donnxruntime_BUILD_SHARED_LIB=ON \
  -DONNX_CUSTOM_PROTOC_EXECUTABLE=/home/lievens/Documents/temp/protoc-3.21.12/bin/protoc \
  -Donnxruntime_BUILD_UNIT_TESTS=OFF \
  ../cmake
make -j6
mayeut commented 1 year ago

for aarch64, adding

SET(CMAKE_SYSTEM_PROCESSOR aarch64)

to the toolchain file did the trick for me with onnxruntime 1.15.1

snnn commented 1 year ago

Thanks for the information. According to cmake's document, the "CMAKE_SYSTEM_PROCESSOR" setting is a must. I will update our document and close this issue.

hfwanguanghui commented 1 year ago

aarch64

in the newest rep 。add ”SET(CMAKE_SYSTEM_PROCESSOR aarch64) “ in toolchain 。the error did not disappear。

... /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:143:18: error: ‘cpuinfo_get_uarchs_count’ was not declared in this scope 143 | ishybrid = cpuinfo_get_uarchs_count() > 1; | ^~~~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:144:25: error: ‘cpuinfo_has_arm_neon_dot’ was not declared in this scope 144 | has_arm_neondot = cpuinfo_has_arm_neon_dot(); | ^~~~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:145:17: error: ‘cpuinfo_has_arm_neon_fp16_arith’ was not declared in this scope 145 | hasfp16 = cpuinfo_has_arm_neon_fp16_arith(); | ^~~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:146:31: error: ‘cpuinfo_get_cores_count’ was not declared in this scope 146 | const uint32_t core_cnt = cpuinfo_get_cores_count(); | ^~~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:150:46: error: ‘cpuinfo_get_processor’ was not declared in this scope; did you mean ‘cpuinfo_processor’? 150 | const struct cpuinfo_processor proc = cpuinfo_get_processor(c); | ^~~~~ | cpuinfo_processor /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:154:46: error: invalid use of incomplete type ‘const struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_processor’ 154 | const struct cpuinfo_core corep = proc->core; ...

but in the version 1.15.1。 it works。

snnn commented 1 year ago

I think they messed up a bit their CMake build environment, they made some changes in that external deps folder.

Because now by default we use find_package to locate the dependencies first. You can set FETCHCONTENT_TRY_FIND_PACKAGE_MODE to NEVER.

cyrusbehr commented 1 year ago

Adding SET(CMAKE_SYSTEM_PROCESSOR aarch64) indeed fixes the issue for v1.15.1. Thank you @mayeut.

For anyone else who comes across this issue, here is the full solution:

Download correct version of protocol buffers

wget https://github.com/protocolbuffers/protobuf/releases/download/v21.12/protoc-21.12-linux-x86_64.zip
unzip protoc-21.12-linux-x86_64.zip -d protoc-3.21.12
rm -rf protoc-21.12-linux-x86_64.zip

Create a file called aarch64.cmake with the following contents:

SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
SET(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
SET(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
SET(CMAKE_SYSTEM_PROCESSOR aarch64)

Now run the following. Be sure to change the path ONNX_CUSTOM_PROTOC_EXECUTABLE argument to point to your protoc download:

git clone --recursive https://github.com/Microsoft/onnxruntime -b v1.15.1 onnxruntime-v1.15.1
cd onnxruntime-v1.15.1
mkdir build
cd build
cmake -Donnxruntime_GCC_STATIC_CPP_RUNTIME=ON -DCMAKE_BUILD_TYPE=Release -Dprotobuf_WITH_ZLIB=OFF -DCMAKE_TOOLCHAIN_FILE=../../aarch64.cmake -Donnxruntime_ENABLE_PYTHON=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_DEV_MODE=OFF -DONNX_CUSTOM_PROTOC_EXECUTABLE=/home/cyrus/work/tmp/protoc-3.21.12/bin/protoc ../cmake/
make -j $(nproc)
cyrusbehr commented 1 year ago

@snnn please be sure to add SET(CMAKE_SYSTEM_PROCESSOR aarch64) to the documentation, it's still not there: image

Feel free to close this issue once you've added it to the documentation page.

csukuangfj commented 11 months ago

aarch64

in the newest rep 。add ”SET(CMAKE_SYSTEM_PROCESSOR aarch64) “ in toolchain 。the error did not disappear。

... /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:143:18: error: ‘cpuinfo_get_uarchs_count’ was not declared in this scope 143 | ishybrid = cpuinfo_get_uarchs_count() > 1; | ^~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:144:25: error: ‘cpuinfo_has_arm_neon_dot’ was not declared in this scope 144 | has_arm_neondot = cpuinfo_has_arm_neon_dot(); | ^~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:145:17: error: ‘cpuinfo_has_arm_neon_fp16_arith’ was not declared in this scope 145 | hasfp16 = cpuinfo_has_arm_neon_fp16_arith(); | ^~~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:146:31: error: ‘cpuinfo_get_cores_count’ was not declared in this scope 146 | const uint32_t core_cnt = cpuinfo_get_cores_count(); | ^~~~~~~ /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:150:46: error: ‘cpuinfo_get_processor’ was not declared in this scope; did you mean ‘cpuinfo_processor’? 150 | const struct cpuinfo_processor proc = cpuinfo_get_processor(c); | ^~~~~ | cpuinfo_processor /mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:154:46: error: invalid use of incomplete type ‘const struct onnxruntime::CPUIDInfo::ArmLinuxInit()::cpuinfo_processor’ 154 | const struct cpuinfo_core corep = proc->core; ...

but in the version 1.15.1。 it works。

I am getting the same error with onnxruntime 1.16.0.

Is there any update on this issue?

snnn commented 11 months ago

It has been fixed in #13891 .

snnn commented 11 months ago
  1. Please build with “-Donnxruntime_ENABLE_CPUINFO=OFF" if cpuinfo caused trouble there.
  2. As @mayeut mentioned, you may also add "SET(CMAKE_SYSTEM_PROCESSOR aarch64)" to your toolchain.cmake file.
csukuangfj commented 11 months ago

@snnn

Thanks for the reply.

For the error,

/mnt/fileroot/guanghui.wan/onnxruntime/onnxruntime/core/common/cpuid_info.cc:144:25: error: ‘cpuinfo_has_arm_neon_dot’ was not declared in this scope 144 | has_arm_neondot = cpuinfo_has_arm_neon_dot(); | ^~~~~~~~

If we use -Donnxruntime_ENABLE_CPUINFO=OFF, then cpuinfo won't be available, but the function cpuinfo_has_arm_neon_dot() is from cpuinfo. So setting -Donnxruntime_ENABLE_CPUINFO=OFF does not fix the issue.

I am building for 32-bit arm, so SET(CMAKE_SYSTEM_PROCESSOR aarch64) also does not help.

SET(CMAKE_SYSTEM_PROCESSOR arm) also does not help.

After digging into the code, I find the following line https://github.com/microsoft/onnxruntime/blob/efef6407bcdd2464f7f38b09579fc6aeb331655e/cmake/external/onnxruntime_external_deps.cmake#L264

so it expects a string matching

armv[5-8].*

Finally, I use SET(CMAKE_SYSTEM_PROCESSOR armv7) and it fixes the issue.

Myzhencai commented 4 months ago

Hi，I just followed the instruction here and seems I get the error bellow:(arm-linux-gnu-gcc v7.5 the same for c++ compiler):

onnxruntime/core/providers/cpu/tensor/upsample.cc:1009:56: error: assuming signed overflow does not occur when assuming that (X + c) < X is always false [-Werror=strict-overflow] for (int64_t i = 0, y_val = y_int - 1; y_val <= y_int + 2; y_val++, i++) {

image

can anyone help with me ,thanks a lot in advance:-)

snnn commented 4 months ago

Can you try GCC 13 instead?

Myzhencai commented 4 months ago

ok thanks a lot

csukuangfj commented 4 months ago

FYI: We provide pre-built shared and static libs of onnxruntime for arm32 and arm64 at https://github.com/csukuangfj/onnxruntime-libs/releases