search

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.3k stars 1.48k forks source link

ERROR: failed to solve: process "/bin/sh -c ./build.sh ${COMMON_BUILD_ARGS} --update --build --allow_running_as_root --use_openvino CPU_FP32" did not complete successfully: #6613

Open vkadlec opened 11 months ago

vkadlec commented 11 months ago

Hi, I am trying to build Triton Interference Server for CPU-only but I have error during comparing values in eigen-3.4..zip

I am using branch 23.09...

Does anyone know what to do, please?

Thanks!!

-- Populating eigen -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /workspace/build/Release/_deps/eigen-subbuild [ 11%] Creating directories for 'eigen-populate' [ 22%] Performing download step (download, verify and extract) for 'eigen-populate' -- Downloading... dst='/workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip' timeout='none' inactivity timeout='none' -- Using src='https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip' -- verifying file... file='/workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip' -- SHA1 hash of /workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip does not match expected value expected: 'ee201b07085203ea7bd8eb97cbcb31b07cfa3efb' actual: '90414f534834cb5041a4c5ddece24978033d87c7' -- Hash mismatch, removing... -- Using src='https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip' -- verifying file... file='/workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip' -- SHA1 hash of /workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip does not match expected value expected: 'ee201b07085203ea7bd8eb97cbcb31b07cfa3efb' actual: '90414f534834cb5041a4c5ddece24978033d87c7' -- Hash mismatch, removing... -- Using src='https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip' -- verifying file... file='/workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip' -- SHA1 hash of /workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip does not match expected value expected: 'ee201b07085203ea7bd8eb97cbcb31b07cfa3efb' actual: '90414f534834cb5041a4c5ddece24978033d87c7' -- Hash mismatch, removing... -- Using src='https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip' -- verifying file... file='/workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip' -- SHA1 hash of /workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip does not match expected value expected: 'ee201b07085203ea7bd8eb97cbcb31b07cfa3efb' actual: '90414f534834cb5041a4c5ddece24978033d87c7' -- Hash mismatch, removing... -- Using src='https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip' -- verifying file... file='/workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip' -- SHA1 hash of /workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip does not match expected value expected: 'ee201b07085203ea7bd8eb97cbcb31b07cfa3efb' actual: '90414f534834cb5041a4c5ddece24978033d87c7' -- Hash mismatch, removing... -- Using src='https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip' -- verifying file... file='/workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip' -- SHA1 hash of /workspace/build/Release/_deps/eigen-subbuild/eigen-populate-prefix/src/eigen-3.4.zip does not match expected value expected: 'ee201b07085203ea7bd8eb97cbcb31b07cfa3efb' actual: '90414f534834cb5041a4c5ddece24978033d87c7' -- Hash mismatch, removing... CMake Error at eigen-subbuild/eigen-populate-prefix/src/eigen-populate-stamp/download-eigen-populate.cmake:170 (message): Each download failed!

gmake[2]: [CMakeFiles/eigen-populate.dir/build.make:100: eigen-populate-prefix/src/eigen-populate-stamp/eigen-populate-download] Error 1 gmake[1]: [CMakeFiles/Makefile2:83: CMakeFiles/eigen-populate.dir/all] Error 2 gmake: *** [Makefile:91: all] Error 2 CMake Error at /usr/share/cmake-3.27/Modules/FetchContent.cmake:1662 (message): Build step for eigen failed: 2 Call Stack (most recent call first): /usr/share/cmake-3.27/Modules/FetchContent.cmake:1802:EVAL:2 (__FetchContent_directPopulate) /usr/share/cmake-3.27/Modules/FetchContent.cmake:1802 (cmake_language) external/eigen.cmake:21 (FetchContent_Populate) external/onnxruntime_external_deps.cmake:428 (include) CMakeLists.txt:549 (include)

-- Configuring incomplete, errors occurred! Traceback (most recent call last): 54 | 55 | >>> RUN ./build.sh ${COMMON_BUILD_ARGS} --update --build --allow_running_as_root --use_openvino CPU_FP32 56 | 57 | #

ERROR: failed to solve: process "/bin/sh -c ./build.sh ${COMMON_BUILD_ARGS} --update --build --allow_running_as_root --use_openvino CPU_FP32" did not complete successfully: exit code: 1 make[2]: [CMakeFiles/ort_target.dir/build.make:74: onnxruntime/lib/libonnxruntime.so] Error 1 make[2]: Leaving directory '/tmp/tritonbuild/onnxruntime/build' make[1]: [CMakeFiles/Makefile2:275: CMakeFiles/ort_target.dir/all] Error 2 make[1]: Leaving directory '/tmp/tritonbuild/onnxruntime/build' make: *** [Makefile:136: all] Error 2

kthui commented 11 months ago

Hi @vkadlec, the hash for eigen-3.4.zip file does not match the one expected. You might want to make sure the file downloaded is the correct one.

Alternatively, you can also use our pre-build container, to skip over any build issues: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags

vkadlec commented 11 months ago

Alternatively, you can also use our pre-build container, to skip over any build issues:

Thank you very much for the answer!

Maybe I should also ask which of the available containers is optimized to run on CPU-only? That means without unnecessary packeges. Is it the xx.yy-py3?

kthui commented 11 months ago

Yes, nvcr.io/nvidia/tritonserver:23.10-py3, for example, is the right container.

Jackiexiao commented 11 months ago

I have the same problem, the hash for eigen-3.4.zip file does not match the one expected. but why.... should we change hash in build.py?

https://gitlab.com/libeigen/eigen/-/archive/3.4/eigen-3.4.zip

nvcr.io/nvidia/tritonserver:23.10-py3 contain gpu/cuda, so I still need to build cpu only image

My script (branch: v23.10)

    python3 build.py  \
        --enable-logging --enable-stats --enable-tracing --enable-metrics \
        --endpoint=http --endpoint=grpc \
        --backend=ensemble \
        --backend=python \
        --backend=onnxruntime
Jackiexiao commented 11 months ago

ok... onnx had fixed this issue https://github.com/microsoft/onnxruntime/issues/18286

use latest triton server code instead of v23.10 / v23.09 fix the issue

docker pull jackiexiao/tritonserver:23.12dev-onnx-py-cpu

vkadlec commented 11 months ago

ok... onnx had fixed this issue microsoft/onnxruntime#18286

use latest triton server code instead of v23.10 / v23.09 fix the issue

docker pull jackiexiao/tritonserver:23.12dev-onnx-py-cpu

thank you! now it is working!

additionally, I would like to ask if the latest code is on the main branch? based on your image image name (23.12dev) I did not find this branch

Jackiexiao commented 11 months ago

yes, it is the latest code is on the main branch, I also build 23.10, only change onnxruntime from 1.16.0 to 1.16.3

docker pull jackiexiao/tritonserver:23.10-onnx-py-cpu

build command

    python3 build.py  \
        --enable-logging --enable-stats --enable-tracing --enable-metrics --enable-cpu-metrics \
        --cache=local --cache=redis \
        --endpoint=http --endpoint=grpc \
        --backend=ensemble \
        --backend=python \
        --backend=onnxruntime
Jackiexiao commented 11 months ago

or you can change image

dyastremsky commented 8 months ago

Closing issue due to inactivity. Please reopen if you would like to follow up with this issue.