Failed to build CUDA docker image[Build]

[huangrunqian]

Describe the issue

I tried to build CUDA docker image from dockerfile in https://github.com/microsoft/onnxruntime/tree/main/dockerfiles, I got failure. How should I fix it?

Urgency

No response

Target platform

Ubuntu 20.04.4 LTS

Build script

git clone https://github.com/Microsoft/onnxruntime.git cd onnxruntime/dockerfiles git submodule update --init docker build -t onnxruntime-cuda -f Dockerfile.cuda ..

Error / output

[ 98%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/providers/internal_testing/internal_testing_tests.cc.o [ 98%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/quantization/quantization_test.cc.o [ 98%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/test_session_options.cc.o [ 98%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/test_run_options.cc.o [ 98%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/test_allocator.cc.o [100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/test_nontensor_types.cc.o [100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/test_model_loading.cc.o [100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/test_ort_format_models.cc.o [100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/utils.cc.o [100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/custom_op_utils.cc.o [100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/shared_lib/test_inference.cc.o [100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/code/onnxruntime/test/unittest_main/test_main.cc.o [100%] Linking CXX executable onnxruntime_test_all [100%] Built target onnxruntime_test_all 2023-02-22 07:14:58,672 util.run [DEBUG] - Subprocess completed. Return code: 0 2023-02-22 07:14:58,673 util.run [INFO] - Running subprocess in '/code/build/Linux/Release' /usr/bin/python3 /code/setup.py bdist_wheel --wheel_name_suffix=gpu Traceback (most recent call last): File "/code/setup.py", line 19, in from wheel.vendored.packaging.tags import sys_tags ModuleNotFoundError: No module named 'wheel.vendored' Traceback (most recent call last): File "/code/tools/ci_build/build.py", line 2744, in sys.exit(main()) File "/code/tools/ci_build/build.py", line 2687, in main build_python_wheel( File "/code/tools/ci_build/build.py", line 2050, in build_python_wheel run_subprocess(args, cwd=cwd) File "/code/tools/ci_build/build.py", line 714, in run_subprocess return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env) File "/code/tools/python/util/run.py", line 49, in run completed_process = subprocess.run( File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/bin/python3', '/code/setup.py', 'bdist_wheel', '--wheel_name_suffix=gpu']' returned non-zero exit status 1. The command '/bin/sh -c cd /code && /bin/bash ./build.sh --skip_submodule_sync --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu/ --use_cuda --config Release --build_wheel --update --build --parallel --cmake_extra_defines ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) 'CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;86'' returned a non-zero code: 1

Visual Studio Version

No response

GCC / Compiler Version

gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

[jack-gits]

I got the same error.

[jack-gits]

`------ Dockerfile.cuda:14

12 |
13 | ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH} 14 | >>> RUN apt-get update && apt-get install -y --no-install-recommends python3-dev ca-certificates g++ python3-numpy gcc make git python3-setuptools python3-wheel python3-packaging python3-pip aria2 && aria2c -q -d /tmp -o cmake-3.24.3-linux-x86_64.tar.gz https://github.com/Kitware/CMake/releases/download/v3.24.3/cmake-3.24.3-linux-x86_64.tar.gz && tar -zxf /tmp/cmake-3.24.3-linux-x86_64.tar.gz --strip=1 -C /usr 15 |
16 | RUN cd /code && /bin/bash ./build.sh --skip_submodule_sync --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu/ --use_cuda --config Release --build_wheel --update --build --parallel --cmake_extra_defines ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) 'CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;86'

ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y --no-install-recommends python3-dev ca-certificates g++ python3-numpy gcc make git python3-setuptools python3-wheel python3-packaging python3-pip aria2 && aria2c -q -d /tmp -o cmake-3.24.3-linux-x86_64.tar.gz https://github.com/Kitware/CMake/releases/download/v3.24.3/cmake-3.24.3-linux-x86_64.tar.gz && tar -zxf /tmp/cmake-3.24.3-linux-x86_64.tar.gz --strip=1 -C /usr" did not complete successfully: exit code: 2`

[zhanghuanrong]

seems not a quantization issue, remove quantization tag.

[Geekgineer]

You need to install "wheel>=0.35.1"

$ pip install "wheel>=0.35.1"

[VeeranjaneyuluToka]

I tried with wheel version 0.41.2, but that did not help me to solve the above issue. Anybody has any solution for this issue yet?