search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.14k stars 2.85k forks source link

Unable to build onnxruntime with "--build_wheel" and "--enable_pybind" options #6841

Open ashutoshnaik opened 3 years ago

ashutoshnaik commented 3 years ago

Hi,

I am unable to build onnxruntime with "--build_wheel" and "--enable_pybind" options. I have confirmed that I have python 3.8 installed and all the dependencies. It compiles perfectly without the above two options. Let me know if I am missing anything

Here is the log:

root@ash-Inspiron-3576:/home/ash/CHRIS_JSON/onnxruntime# ./build.sh --config Release --build_wheel --enable_pybind 
2021-02-27 18:19:12,174 build [DEBUG] - Defaulting to running update, build [and test for native builds].
2021-02-27 18:19:12,174 build [INFO] - Build started
2021-02-27 18:19:12,175 util.run [INFO] - Running subprocess in '/home/ash/CHRIS_JSON/onnxruntime'
['git', 'submodule', 'sync', '--recursive']
Synchronizing submodule url for 'cmake/external/FeaturizersLibrary'
Synchronizing submodule url for 'cmake/external/FeaturizersLibrary/src/3rdParty/eigen'
Synchronizing submodule url for 'cmake/external/FeaturizersLibrary/src/3rdParty/re2'
Synchronizing submodule url for 'cmake/external/SafeInt/safeint'
Synchronizing submodule url for 'cmake/external/coremltools'
Synchronizing submodule url for 'cmake/external/cub'
Synchronizing submodule url for 'cmake/external/cxxopts'
Synchronizing submodule url for 'cmake/external/date'
Synchronizing submodule url for 'cmake/external/eigen'
Synchronizing submodule url for 'cmake/external/flatbuffers'
Synchronizing submodule url for 'cmake/external/googletest'
Synchronizing submodule url for 'cmake/external/json'
Synchronizing submodule url for 'cmake/external/libprotobuf-mutator'
Synchronizing submodule url for 'cmake/external/mimalloc'
Synchronizing submodule url for 'cmake/external/mp11'
Synchronizing submodule url for 'cmake/external/nsync'
Synchronizing submodule url for 'cmake/external/onnx'
Synchronizing submodule url for 'cmake/external/onnx/third_party/benchmark'
Synchronizing submodule url for 'cmake/external/onnx/third_party/pybind11'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt/third_party/onnx'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt/third_party/onnx/third_party/benchmark'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt/third_party/onnx/third_party/pybind11'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
Synchronizing submodule url for 'cmake/external/optional-lite'
Synchronizing submodule url for 'cmake/external/protobuf'
Synchronizing submodule url for 'cmake/external/protobuf/third_party/benchmark'
Synchronizing submodule url for 'cmake/external/protobuf/third_party/googletest'
Synchronizing submodule url for 'cmake/external/re2'
Synchronizing submodule url for 'cmake/external/tensorboard'
Synchronizing submodule url for 'cmake/external/tvm'
Synchronizing submodule url for 'cmake/external/tvm/3rdparty/HalideIR'
Synchronizing submodule url for 'cmake/external/tvm/3rdparty/dlpack'
Synchronizing submodule url for 'cmake/external/tvm/3rdparty/dmlc-core'
Synchronizing submodule url for 'cmake/external/tvm/3rdparty/rang'
Synchronizing submodule url for 'cmake/external/wil'
Synchronizing submodule url for 'server/external/spdlog'
2021-02-27 18:19:12,447 util.run [DEBUG] - Subprocess completed. Return code: 0
2021-02-27 18:19:12,447 util.run [INFO] - Running subprocess in '/home/ash/CHRIS_JSON/onnxruntime'
['git', 'submodule', 'update', '--init', '--recursive']
2021-02-27 18:19:13,040 util.run [DEBUG] - Subprocess completed. Return code: 0
2021-02-27 18:19:13,040 build [INFO] - Generating CMake build tree
2021-02-27 18:19:13,041 util.run [INFO] - Running subprocess in '/home/ash/CHRIS_JSON/onnxruntime/build/Linux/Release'
['/usr/bin/cmake', '/home/ash/CHRIS_JSON/onnxruntime/cmake', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_BUILD_WINML_TESTS=ON', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-Donnxruntime_DEV_MODE=ON', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-Donnxruntime_USE_CUDA=OFF', '-Donnxruntime_CUDA_VERSION=', '-Donnxruntime_CUDA_HOME=', '-Donnxruntime_CUDNN_HOME=', '-Donnxruntime_USE_FEATURIZERS=OFF', '-Donnxruntime_USE_MIMALLOC_STL_ALLOCATOR=OFF', '-Donnxruntime_USE_MIMALLOC_ARENA_ALLOCATOR=OFF', '-Donnxruntime_ENABLE_PYTHON=ON', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=OFF', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_SHARED_LIB=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_DNNL_GPU_RUNTIME=', '-Donnxruntime_DNNL_OPENCL_ROOT=', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_OPENMP=OFF', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_NUPHAR=OFF', '-Donnxruntime_USE_TENSORRT=OFF', '-Donnxruntime_TENSORRT_HOME=', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_MIGRAPHX_HOME=', '-Donnxruntime_CROSS_COMPILING=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_DISABLE_ORT_FORMAT_LOAD=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_MSVC_STATIC_RUNTIME=OFF', '-Donnxruntime_ENABLE_LANGUAGE_INTEROP_OPS=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ACL_1902=OFF', '-Donnxruntime_USE_ACL_1905=OFF', '-Donnxruntime_USE_ACL_1908=OFF', '-Donnxruntime_USE_ACL_2002=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=ON', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-Donnxruntime_ROCM_HOME=', '-DOnnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=ON', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_PYBIND_EXPORT_OPSCHEMA=OFF', '-Donnxruntime_ENABLE_MEMLEAK_CHECKER=OFF', '-DCMAKE_BUILD_TYPE=Release']
F16C instruction set is not supported.
FMA instruction set is not supported.
AVX instruction set is not supported.
One or more AVX/F16C instruction flags are not supported. 
Use gtest from submodule
-- Found PythonInterp: /usr/bin/python3 (found version "3.8.5") 
Doing crosscompiling
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.5", minimum required is "3.5") 
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
  (Required is at least version "3.5")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindPythonLibs.cmake:310 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:554 (find_package)

-- Configuring incomplete, errors occurred!
See also "/home/ash/CHRIS_JSON/onnxruntime/build/Linux/Release/CMakeFiles/CMakeOutput.log".
See also "/home/ash/CHRIS_JSON/onnxruntime/build/Linux/Release/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "/home/ash/CHRIS_JSON/onnxruntime/tools/ci_build/build.py", line 2003, in <module>
    sys.exit(main())
  File "/home/ash/CHRIS_JSON/onnxruntime/tools/ci_build/build.py", line 1923, in main
    generate_build_tree(
  File "/home/ash/CHRIS_JSON/onnxruntime/tools/ci_build/build.py", line 963, in generate_build_tree
    run_subprocess(
  File "/home/ash/CHRIS_JSON/onnxruntime/tools/ci_build/build.py", line 536, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/home/ash/CHRIS_JSON/onnxruntime/tools/python/util/run.py", line 41, in run
    completed_process = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/cmake', '/home/ash/CHRIS_JSON/onnxruntime/cmake', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_BUILD_WINML_TESTS=ON', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-Donnxruntime_DEV_MODE=ON', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-Donnxruntime_USE_CUDA=OFF', '-Donnxruntime_CUDA_VERSION=', '-Donnxruntime_CUDA_HOME=', '-Donnxruntime_CUDNN_HOME=', '-Donnxruntime_USE_FEATURIZERS=OFF', '-Donnxruntime_USE_MIMALLOC_STL_ALLOCATOR=OFF', '-Donnxruntime_USE_MIMALLOC_ARENA_ALLOCATOR=OFF', '-Donnxruntime_ENABLE_PYTHON=ON', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=OFF', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_SHARED_LIB=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_DNNL_GPU_RUNTIME=', '-Donnxruntime_DNNL_OPENCL_ROOT=', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_OPENMP=OFF', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_NUPHAR=OFF', '-Donnxruntime_USE_TENSORRT=OFF', '-Donnxruntime_TENSORRT_HOME=', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_MIGRAPHX_HOME=', '-Donnxruntime_CROSS_COMPILING=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_DISABLE_ORT_FORMAT_LOAD=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_MSVC_STATIC_RUNTIME=OFF', '-Donnxruntime_ENABLE_LANGUAGE_INTEROP_OPS=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ACL_1902=OFF', '-Donnxruntime_USE_ACL_1905=OFF', '-Donnxruntime_USE_ACL_1908=OFF', '-Donnxruntime_USE_ACL_2002=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=ON', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-Donnxruntime_ROCM_HOME=', '-DOnnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=ON', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_PYBIND_EXPORT_OPSCHEMA=OFF', '-Donnxruntime_ENABLE_MEMLEAK_CHECKER=OFF', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

Thanks Ashutosh

jcwchen commented 3 years ago

Hi @ashutoshnaik, On some platforms with installed Python, PythonLibs from Cmake cannot find the right Python path. What's your environment?Could you try apt-get install python3-dev before the ORT installation? Thank you.

ashutoshnaik commented 3 years ago

Hi @jcwchen ,

I am using Ubuntu 20.04. I installed cmake 3.16.3 from sources.

python3-dev is already installed and is the latest version :

root@ash-Inspiron-3576:/home/ash# apt-get install python3-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-dev is already the newest version (3.8.2-0ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded.

Do let me know if I need to reinstall everything, or how can I proceed ?

Thanks, Ashutosh Naik.

edgchen1 commented 3 years ago

Some other things to try:

Use a virtual Python environment instead of the system Python, like conda or python3-venv. This has the additional benefit of not requiring or installing (which build.py automatically does in some cases) onnxruntime dependencies in your system Python installation. I didn't see this error when using a virtual environment on Ubuntu 20.04.

It is possible to manually set PYTHON_LIBRARY and PYTHON_INCLUDE_DIR (more info at https://cmake.org/cmake/help/latest/module/FindPythonLibs.html). You can give an additional argument to build.sh: --cmake_extra_defines PYTHON_LIBRARY=... PYTHON_INCLUDE_DIR=..., see an example here: https://github.com/microsoft/onnxruntime/blob/5cf66069645d338b481e70dcfe869161c03e102f/tools/ci_build/github/azure-pipelines/linux-gpu-ci-pipeline.yml#L42

And just to note, the --build_wheel option implies --enable_pybind.

ashutoshnaik commented 3 years ago

It did work for me by setting additional argument to build.sh:

--cmake_extra_defines PYTHON_LIBRARY=... PYTHON_INCLUDE_DIR=...,

However, this could probably be solved in a better way ?