search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.73k stars 2.94k forks source link

[Build] cuda dockerfile build error #21792

Closed hassan-twelvelabs closed 2 months ago

hassan-twelvelabs commented 3 months ago

Describe the issue

Please see the error message attached.

Urgency

Urgent; broken docker image build.

Target platform

cuda

Build script

docker build -t onnxruntime-cuda -f Dockerfile.cuda ..

Error / output


897.6 copying onnxruntime/Privacy.md -> build/lib/onnxruntime
897.6 copying onnxruntime/tools/mobile_helpers/nnapi_supported_ops.md -> build/lib/onnxruntime/tools/mobile_helpers
897.6 copying onnxruntime/tools/mobile_helpers/coreml_supported_neuralnetwork_ops.md -> build/lib/onnxruntime/tools/mobile_helpers
897.6 copying onnxruntime/tools/mobile_helpers/coreml_supported_mlprogram_ops.md -> build/lib/onnxruntime/tools/mobile_helpers
897.7 /usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
897.7 !!
897.7 
897.7         ********************************************************************************
897.7         Please avoid running ``setup.py`` directly.
897.7         Instead, use pypa/build, pypa/installer or other
897.7         standards-based tools.
897.7 
897.7         See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
897.7         ********************************************************************************
897.7 
897.7 !!
897.7   self.initialize_options()
897.7 installing to build/bdist.linux-x86_64/wheel
897.7 running install
897.7 Traceback (most recent call last):
897.7   File "/code/setup.py", line 750, in <module>
897.7     setup(
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 108, in setup
897.7     return distutils.core.setup(**attrs)
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 184, in setup
897.7     return run_commands(dist)
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 200, in run_commands
897.7     dist.run_commands()
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 964, in run_commands
897.7     self.run_command(cmd)
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
897.7     super().run_command(command)
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 983, in run_command
897.7     cmd_obj.run()
897.7   File "/code/setup.py", line 280, in run
897.7     _bdist_wheel.run(self)
897.7   File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 335, in run
897.7     self.run_command('install')
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
897.7     self.distribution.run_command(command)
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 945, in run_command
897.7     super().run_command(command)
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 982, in run_command
897.7     cmd_obj.ensure_finalized()
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
897.7     self.finalize_options()
897.7   File "/code/setup.py", line 303, in finalize_options
897.7     ret = InstallCommandBase.finalize_options(self)
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/command/install.py", line 62, in finalize_options
897.7     super().finalize_options()
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/install.py", line 407, in finalize_options
897.7     'dist_fullname': self.distribution.get_fullname(),
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 266, in get_fullname
897.7     return _distribution_fullname(self.get_name(), self.get_version())
897.7   File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
897.7     canonicalize_version(version, strip_trailing_zero=False),
897.7 TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
897.7 Traceback (most recent call last):
897.7   File "/code/tools/ci_build/build.py", line 2971, in <module>
897.7 Namespace(build_dir='/code/build/Linux', config=['Release'], update=True, build=True, clean=False, parallel=0, nvcc_threads=-1, test=False, skip_tests=False, compile_no_warning_as_error=False, enable_nvtx_profile=False, enable_memory_profile=False, enable_training=False, enable_training_apis=False, enable_training_ops=False, enable_nccl=False, mpi_home=None, nccl_home=None, use_mpi=False, enable_onnx_tests=False, path_to_protoc_exe=None, fuzz_testing=False, enable_symbolic_shape_infer_tests=False, gen_doc=None, gen_api_doc=False, use_cuda=True, cuda_version=None, cuda_home='/usr/local/cuda', cudnn_home='/usr/lib/x86_64-linux-gnu/', enable_cuda_line_info=False, enable_cuda_nhwc_ops=False, enable_pybind=False, build_wheel=True, wheel_name_suffix=None, skip_keras_test=False, build_csharp=False, build_nuget=False, msbuild_extra_options=None, build_java=False, build_nodejs=False, build_objc=False, build_shared_lib=False, build_apple_framework=False, cmake_extra_defines=[['ONNXRUNTIME_VERSION=1.20.0', 'CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;86']], target=None, x86=False, rv64=False, arm=False, arm64=False, arm64ec=False, buildasx=False, riscv_toolchain_root='', riscv_qemu_path='', msvc_toolset=None, windows_sdk_version=None, android=False, android_abi='arm64-v8a', android_api=27, android_sdk_path='', android_ndk_path='', android_cpp_shared=False, android_run_emulator=False, use_gdk=False, gdk_edition='.', gdk_platform='Scarlett', ios=False, visionos=False, macos=None, apple_sysroot='', ios_toolchain_file='', visionos_toolchain_file='', xcode_code_signing_team_id='', xcode_code_signing_identity='', cmake_generator=None, osx_arch='x86_64', apple_deploy_target=None, enable_address_sanitizer=False, use_binskim_compliant_compile_flags=False, disable_memleak_checker=False, build_wasm=False, build_wasm_static_lib=False, emsdk_version='3.1.62', enable_wasm_simd=False, enable_wasm_threads=False, disable_wasm_exception_catching=False, enable_wasm_api_exception_catching=False, enable_wasm_exception_throwing_override=True, wasm_run_tests_in_browser=False, enable_wasm_profiling=False, enable_wasm_debug_info=False, wasm_malloc=None, emscripten_settings=None, use_extensions=False, extensions_overridden_path=None, cmake_path='cmake', ctest_path='ctest', skip_submodule_sync=True, use_mimalloc=False, use_dnnl=False, dnnl_gpu_runtime='', dnnl_opencl_root='', use_openvino=None, dnnl_aarch64_runtime='', dnnl_acl_root='', use_coreml=False, use_webnn=False, use_snpe=False, snpe_root=None, use_nnapi=False, use_vsinpu=False, nnapi_min_api=None, use_jsep=False, use_qnn=False, qnn_home=None, use_rknpu=False, use_preinstalled_eigen=False, eigen_path=None, enable_msinternal=False, llvm_path=None, use_vitisai=False, use_tvm=False, tvm_cuda_runtime=False, use_tvm_hash=False, use_tensorrt=False, use_tensorrt_builtin_parser=True, use_tensorrt_oss_parser=False, tensorrt_home=None, test_all_timeout='10800', use_migraphx=False, migraphx_home=None, use_full_protobuf=False, llvm_config='', skip_onnx_tests=False, skip_winml_tests=False, skip_nodejs_tests=False, enable_msvc_static_runtime=False, use_dml=False, dml_path='', use_winml=False, winml_root_namespace_override=None, dml_external_project=False, use_telemetry=False, enable_wcos=False, enable_lto=False, enable_transformers_tool_test=False, use_acl=None, acl_home=None, acl_libs=None, use_armnn=False, armnn_relu=False, armnn_bn=False, armnn_home=None, armnn_libs=None, build_micro_benchmarks=False, minimal_build=None, include_ops_by_config=None, enable_reduced_operator_type_support=False, disable_contrib_ops=False, disable_ml_ops=False, disable_rtti=False, disable_types=[], disable_exceptions=False, rocm_version=None, use_rocm=False, rocm_home=None, code_coverage=False, enable_lazy_tensor=False, ms_experimental=False, enable_external_custom_op_schemas=False, external_graph_transformer_path=None, enable_cuda_profiling=False, use_cann=False, cann_home=None, enable_rocm_profiling=False, use_xnnpack=False, use_azure=False, use_cache=False, use_triton_kernel=False, use_lock_free_queue=False, allow_running_as_root=True)
897.7 Failed to import psutil. Please `pip install psutil` for better estimation of nvcc threads. Use nvcc_threads=1
897.7     sys.exit(main())
897.7   File "/code/tools/ci_build/build.py", line 2891, in main
897.7     build_python_wheel(
897.7   File "/code/tools/ci_build/build.py", line 2284, in build_python_wheel
897.7     run_subprocess(args, cwd=cwd)
897.7   File "/code/tools/ci_build/build.py", line 860, in run_subprocess
897.7     return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
897.7   File "/code/tools/python/util/run.py", line 49, in run
897.7     completed_process = subprocess.run(
897.7   File "/usr/lib/python3.10/subprocess.py", line 526, in run
897.7     raise CalledProcessError(retcode, process.args,
897.7 subprocess.CalledProcessError: Command '['/usr/bin/python3', '/code/setup.py', 'bdist_wheel', '--wheel_name_suffix=gpu']' returned non-zero exit status 1.
------
Dockerfile.cuda:16
--------------------
  15 |     
  16 | >>> RUN cd /code && python3 -m pip install -r tools/ci_build/github/linux/docker/inference/x86_64/python/cuda/scripts/requireme\
  17 | >>> nts.txt && /bin/bash ./build.sh --allow_running_as_root --skip_submodule_sync --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu/ --use_cuda --config Release --build_wheel --update --build --parallel --cmake_extra_defines ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) 'CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;86'
  18 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /code && python3 -m pip install -r tools/ci_build/github/linux/docker/inference/x86_64/python/cuda/scripts/requirements.txt && /bin/bash ./build.sh --allow_running_as_root --skip_submodule_sync --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu/ --use_cuda --config Release --build_wheel --update --build --parallel --cmake_extra_defines ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) 'CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;86'" did not complete successfully: exit code: 1```

### Visual Studio Version

_No response_

### GCC / Compiler Version

_No response_
mindest commented 3 months ago

@hassan-twelvelabs, please retry with the above change to see if it is resolved.

hassan-twelvelabs commented 2 months ago

@hassan-twelvelabs, please retry with the above change to see if it is resolved.

Thank you , yes, that solved the issue for me!

mindest commented 2 months ago

Thanks for confirming!