search

triton-inference-server / fil_backend

FIL backend for the Triton Inference Server
Apache License 2.0
72 stars 36 forks source link

build fil_backend with docker encountered cmake error: rapids-cmake,rapids-cpm,rapids-cuda,rapids-export,rapids-find not found #380

Closed sunjiabin17 closed 10 months ago

sunjiabin17 commented 10 months ago

Hi, I'm trying to build fil backend from the triton-inference-server/server build.py generated script. I built the tritonserver_builder image, and enter it to run cmake build manually because of some network issues caused failures. when I trying to build fil backend, i got an error while executing cmake --build . --config Debug -j24 -v -t install

FYI I pulled commit 8f9878966468664b39263db7f7d0dbb09db09ecb from triton-inference-server/server, and run ./build.py --build-type Debug --dryrun --enable-all -v to generate build script and dockerfile;

I did small changes so I can build it using my host network and proxy;

I had built triton inference server debug type successfully a few months ago, I don't why this failure happens.

  1. CMakeLists.txt:34 https://github.com/triton-inference-server/fil_backend/blob/e988867595991221e4368b8fa697567a536cb0c8/CMakeLists.txt#L34

to COMMAND DOCKER_BUILDKIT=1 docker build --network=host -t

  1. ops/Dockerfile:76 https://github.com/triton-inference-server/fil_backend/blob/e988867595991221e4368b8fa697567a536cb0c8/ops/Dockerfile#L76

to && curl -x http://127.0.0.1:7890 -o /tmp/cuda-keyring.deb \

here is my build cmds

mkdir -p /tmp/tritonbuild
cd /tmp/tritonbuild
rm -fr openvino
if [[ ! -e openvino ]]; then
  git clone --recursive --single-branch --depth=1 -b main https://github.com/triton-inference-server/openvino_backend.git openvino;
fi
mkdir -p /tmp/tritonbuild/openvino/build
cd /tmp/tritonbuild/openvino/build
cmake "-DTRT_VERSION=${TRT_VERSION}" "-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}" "-DVCPKG_TARGET_TRIPLET=${VCPKG_TARGET_TRIPLET}" "-DTRITON_BUILD_OPENVINO_VERSION=2023.0.0" "-DTRITON_BUILD_CONTAINER_VERSION=24.01" "-DCMAKE_BUILD_TYPE=Debug" "-DCMAKE_INSTALL_PREFIX:PATH=/tmp/tritonbuild/openvino/install" "-DTRITON_COMMON_REPO_TAG:STRING=main" "-DTRITON_CORE_REPO_TAG:STRING=main" "-DTRITON_BACKEND_REPO_TAG:STRING=main" "-DTRITON_ENABLE_GPU:BOOL=ON" "-DTRITON_ENABLE_MALI_GPU:BOOL=OFF" "-DTRITON_ENABLE_STATS:BOOL=ON" "-DTRITON_ENABLE_METRICS:BOOL=ON" "-DTRITON_ENABLE_MEMORY_TRACKER:BOOL=ON" ..
cmake --build . --config Debug -j24 -v -t install
mkdir -p /tmp/tritonbuild/install/backends
rm -fr /tmp/tritonbuild/install/backends/openvino
cp -r /tmp/tritonbuild/openvino/install/backends/openvino /tmp/tritonbuild/install/backends

error full logs

Change Dir: '/tmp/tritonbuild/fil/build'

Run Build Command(s): /usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile -j24 install
/usr/bin/cmake -S/tmp/tritonbuild/fil -B/tmp/tritonbuild/fil/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /tmp/tritonbuild/fil/build/CMakeFiles /tmp/tritonbuild/fil/build//CMakeFiles/progress.marks
/usr/bin/gmake  -f CMakeFiles/Makefile2 all
gmake[1]: Entering directory '/tmp/tritonbuild/fil/build'
/usr/bin/gmake  -f CMakeFiles/fil_docker.dir/build.make CMakeFiles/fil_docker.dir/depend
gmake[2]: Entering directory '/tmp/tritonbuild/fil/build'
cd /tmp/tritonbuild/fil/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/tritonbuild/fil /tmp/tritonbuild/fil /tmp/tritonbuild/fil/build /tmp/tritonbuild/fil/build /tmp/tritonbuild/fil/build/CMakeFiles/fil_docker.dir/DependInfo.cmake "--color="
gmake[2]: Leaving directory '/tmp/tritonbuild/fil/build'
/usr/bin/gmake  -f CMakeFiles/fil_docker.dir/build.make CMakeFiles/fil_docker.dir/build
gmake[2]: Entering directory '/tmp/tritonbuild/fil/build'
[100%] Building FIL backend in Docker
DOCKER_BUILDKIT=1 docker build --network=host -t triton_fil_builder --build-arg BUILD_TYPE=Release --build-arg TRITON_VERSION=24.01 --build-arg BASE_IMAGE=nvcr.io/nvidia/tritonserver:23.09-py3 --build-arg TRITON_ENABLE_GPU=ON --build-arg TRITON_ENABLE_STATS=ON --build-arg TRITON_COMMON_REPO_TAG=main --build-arg TRITON_CORE_REPO_TAG=main --build-arg TRITON_BACKEND_REPO_TAG=main -f /tmp/tritonbuild/fil/ops/Dockerfile /tmp/tritonbuild/fil
[+] Building 13.6s (30/32)                                                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                  0.2s
 => => transferring dockerfile: 7.15kB                                                                                                                                                0.0s
 => resolve image config for docker.io/docker/dockerfile:1.3                                                                                                                          2.4s
 => CACHED docker-image://docker.io/docker/dockerfile:1.3@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.3s
 => => transferring context: 103B                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/condaforge/mambaforge:latest                                                                                                               2.5s
 => [internal] load metadata for nvcr.io/nvidia/tritonserver:23.09-py3                                                                                                                2.8s
 => [internal] load build context                                                                                                                                                     0.2s
 => => transferring context: 1.60kB                                                                                                                                                   0.0s
 => [base 1/3] FROM nvcr.io/nvidia/tritonserver:23.09-py3@sha256:70b869e4aeb40b30a6bda8b8e22b38800dd9384c9fbab97934b2b6c24d9aa8e8                                                     0.0s
 => [conda-base 1/6] FROM docker.io/condaforge/mambaforge@sha256:ca48a4426c1642ce916ed55cbb3b06c022d7e38649d0cbb6bce6f876bab8a215                                                     0.0s
 => CACHED [final 2/4] RUN mkdir /models                                                                                                                                              0.0s
 => CACHED [final 3/4] RUN if [ -d /opt/tritonserver/backends/fil ];     then       rm -rf /opt/tritonserver/backends/fil/*;     fi                                                   0.1s
 => CACHED [base 2/3] RUN  if command -v curl;  then [ $(uname -m) = 'x86_64' ]  && curl -x http://127.0.0.1:7890 -o /tmp/cuda-keyring.deb       https://developer.download.nvidia.c  0.0s
 => CACHED [base 3/3] RUN apt-get update     && apt-get install --no-install-recommends -y       build-essential       ca-certificates       git     && apt-get clean     && rm -rf   0.0s
 => CACHED [build-prep 1/6] RUN mkdir -p /rapids_triton/build /rapids_triton/install                                                                                                  0.0s
 => CACHED [build-prep 2/6] COPY ./src /rapids_triton/src                                                                                                                             0.0s
 => CACHED [build-prep 3/6] COPY ./CMakeLists.txt /rapids_triton                                                                                                                      0.0s
 => CACHED [build-prep 4/6] COPY ./cmake /rapids_triton/cmake                                                                                                                         0.0s
 => CACHED [build-prep 5/6] WORKDIR /rapids_triton/build                                                                                                                              0.0s
 => CACHED [build-prep 6/6] RUN if [ -d /opt/tritonserver/backends/fil ];     then       rm -rf /opt/tritonserver/backends/fil/*;     else       mkdir -p /opt/tritonserver/backends  0.0s
 => CACHED [conda-base 2/6] COPY ./ops/gpuci_conda_retry /usr/bin/gpuci_conda_retry                                                                                                   0.0s
 => CACHED [conda-base 3/6] COPY ./ops/gpuci_mamba_retry /usr/bin/gpuci_mamba_retry                                                                                                   0.0s
 => CACHED [conda-base 4/6] RUN chmod +x /usr/bin/gpuci_conda_retry /usr/bin/gpuci_mamba_retry                                                                                        0.0s
 => CACHED [conda-base 5/6] RUN mkdir /conda                                                                                                                                          0.0s
 => CACHED [conda-base 6/6] RUN gpuci_mamba_retry install -c conda-forge conda-pack=0.7                                                                                               0.0s
 => CACHED [conda-dev 1/4] COPY ./conda/environments/rapids_triton_dev.yml /conda/environment.yml                                                                                     0.0s
 => CACHED [conda-dev 2/4] RUN gpuci_mamba_retry env update -f /conda/environment.yml  && rm /conda/environment.yml                                                                   0.0s
 => CACHED [conda-dev 3/4] RUN conda-pack -n rapids_triton_dev -o /tmp/env.tar  && mkdir /conda/dev/  && cd /conda/dev/  && tar xf /tmp/env.tar  && rm /tmp/env.tar                   0.0s
 => CACHED [conda-dev 4/4] RUN /conda/dev/bin/conda-unpack                                                                                                                            0.0s
 => CACHED [build-stage 1/3] COPY --from=conda-dev /conda/dev /conda/dev                                                                                                              0.0s
 => ERROR [build-stage 2/3] RUN source /conda/dev/bin/activate  && cmake       --log-level=VERBOSE       -GNinja       -DCMAKE_BUILD_TYPE="Release"       -DBUILD_TESTS=""       -DT  6.0s
------
 > [build-stage 2/3] RUN source /conda/dev/bin/activate  && cmake       --log-level=VERBOSE       -GNinja       -DCMAKE_BUILD_TYPE="Release"       -DBUILD_TESTS=""       -DTRITON_CORE_REPO_TAG="main"       -DTRITON_COMMON_REPO_TAG="main"       -DTRITON_BACKEND_REPO_TAG="main"       -DTRITON_ENABLE_GPU="ON"       -DTRITON_ENABLE_STATS="ON"       -DRAPIDS_DEPENDENCIES_VERSION="23.12"       -DTRITON_FIL_USE_TREELITE_STATIC="ON"       -DCMAKE_INSTALL_PREFIX=/rapids_triton/install       ..;:
5.432 CMake Error at CMakeLists.txt:76 (include):
5.432   include could not find requested file:
5.432
5.432     rapids-cmake
5.432
5.432
5.432 CMake Error at CMakeLists.txt:77 (include):
5.432   include could not find requested file:
5.432
5.432     rapids-cpm
5.432
5.432
5.432 CMake Error at CMakeLists.txt:78 (include):
5.432   include could not find requested file:
5.432
5.432     rapids-cuda
5.432
5.432
5.433 CMake Error at CMakeLists.txt:79 (include):
5.433   include could not find requested file:
5.433
5.433     rapids-export
5.433
5.433
5.433 CMake Error at CMakeLists.txt:80 (include):
5.433   include could not find requested file:
5.433
5.433     rapids-find
5.433
5.433
5.433 -- RAPIDS_TRITON_BACKEND: Enabling detection of conda environment for dependencies: ON
5.433 -- RAPIDS_TRITON_BACKEND: Enabling kernelinfo in nvcc: OFF
5.433 -- RAPIDS_TRITON_BACKEND: Enabling lineinfo in nvcc: OFF
5.433 -- RAPIDS_TRITON_BACKEND: Enabling nvtx markers: OFF
5.433 -- RAPIDS_TRITON_BACKEND: Build RAPIDS_TRITON_BACKEND unit-tests:
5.433 -- RAPIDS_TRITON_BACKEND: RAPIDS projects dependencies version: 23.12
5.433 -- RAPIDS_TRITON_BACKEND: Linking Treelite statically:
5.433 -- RAPIDS_TRITON_BACKEND: RMM_LOGGING_LEVEL = 'INFO'.
5.433 CMake Error at CMakeLists.txt:113 (rapids_cuda_init_architectures):
5.433   Unknown CMake command "rapids_cuda_init_architectures".
5.433
5.433
5.434 -- Configuring incomplete, errors occurred!
------
Dockerfile:148
--------------------
 147 |
 148 | >>> RUN source /conda/dev/bin/activate \
 149 | >>>  && cmake \
 150 | >>>       --log-level=VERBOSE \
 151 | >>>       -GNinja \
 152 | >>>       -DCMAKE_BUILD_TYPE="${BUILD_TYPE}" \
 153 | >>>       -DBUILD_TESTS="${BUILD_TESTS}" \
 154 | >>>       -DTRITON_CORE_REPO_TAG="${TRITON_CORE_REPO_TAG}" \
 155 | >>>       -DTRITON_COMMON_REPO_TAG="${TRITON_COMMON_REPO_TAG}" \
 156 | >>>       -DTRITON_BACKEND_REPO_TAG="${TRITON_BACKEND_REPO_TAG}" \
 157 | >>>       -DTRITON_ENABLE_GPU="${TRITON_ENABLE_GPU}" \
 158 | >>>       -DTRITON_ENABLE_STATS="${TRITON_ENABLE_STATS}" \
 159 | >>>       -DRAPIDS_DEPENDENCIES_VERSION="${RAPIDS_DEPENDENCIES_VERSION}" \
 160 | >>>       -DTRITON_FIL_USE_TREELITE_STATIC="${TRITON_FIL_USE_TREELITE_STATIC}" \
 161 | >>>       -DCMAKE_INSTALL_PREFIX=/rapids_triton/install \
 162 | >>>       ..;
 163 |
--------------------
ERROR: failed to solve: process "/bin/bash -c source /conda/dev/bin/activate  && cmake       --log-level=VERBOSE       -GNinja       -DCMAKE_BUILD_TYPE=\"${BUILD_TYPE}\"       -DBUILD_TESTS=\"${BUILD_TESTS}\"       -DTRITON_CORE_REPO_TAG=\"${TRITON_CORE_REPO_TAG}\"       -DTRITON_COMMON_REPO_TAG=\"${TRITON_COMMON_REPO_TAG}\"       -DTRITON_BACKEND_REPO_TAG=\"${TRITON_BACKEND_REPO_TAG}\"       -DTRITON_ENABLE_GPU=\"${TRITON_ENABLE_GPU}\"       -DTRITON_ENABLE_STATS=\"${TRITON_ENABLE_STATS}\"       -DRAPIDS_DEPENDENCIES_VERSION=\"${RAPIDS_DEPENDENCIES_VERSION}\"       -DTRITON_FIL_USE_TREELITE_STATIC=\"${TRITON_FIL_USE_TREELITE_STATIC}\"       -DCMAKE_INSTALL_PREFIX=/rapids_triton/install       ..;" did not complete successfully: exit code: 1
gmake[2]: *** [CMakeFiles/fil_docker.dir/build.make:74: fil/libtriton_fil.so] Error 1
gmake[2]: Leaving directory '/tmp/tritonbuild/fil/build'
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/fil_docker.dir/all] Error 2
gmake[1]: Leaving directory '/tmp/tritonbuild/fil/build'
gmake: *** [Makefile:136: all] Error 2
sunjiabin17 commented 10 months ago

it seems like downloading RAPIDS.cmake from raw.githubusercontent.com failed but without error prompts. here https://github.com/triton-inference-server/fil_backend/blob/e988867595991221e4368b8fa697567a536cb0c8/CMakeLists.txt#L71-L74

it goes well after i specified proxy server here https://github.com/triton-inference-server/fil_backend/blob/e988867595991221e4368b8fa697567a536cb0c8/ops/Dockerfile#L149

RUN source /conda/dev/bin/activate \
 && export http_proxy=http://127.0.0.1:7890 && export https_proxy=http://127.0.0.1:7890 && cmake \
...