Closed mujahiddh closed 11 months ago
fpetrini15
commented
11 months ago Hello @mujahiddh, thank you for reporting your issue. As in the issue you referenced, the "No configure step for 'grpc-repo'" message can appear in the logs, even if that is not the source of the issue. Your logs appear to be abridged. Is it possible to post the full log?
mujahiddh
commented
11 months ago Sending build context to Docker daemon 18.24MB
mujahiddh
commented
11 months ago mujahiddh
commented
11 months ago mujahiddh
commented
11 months ago @fpetrini15 i have shared above the logs, as much of it as feasible, some parts i had to skip, because its a lot. hope this give more clarity. on another run, seems the build stops at the above point, not sure why, will appreciate any help on this, thanks
fpetrini15
commented
11 months ago Looks like your error originates from here:
[ 12%] �[32m�[1mLinking CXX shared library triton_bindings.cpython-310-x86_64-linux-gnu.so�[0m cd /tmp/tritonbuild/tritonserver/build/_deps/repo-core-build/python/tritonserver && /usr/bin/cmake -E cmake_link_script CMakeFiles/python-bindings.dir/link.txt --verbose=0 -- Looking for unistd.h - found -- Looking for netdb.h -- Looking for netdb.h - found -- Looking for dlfcn.h make[3]: write jobserver: Bad file descriptor. Stop. make[3]: Waiting for unfinished jobs.... make[3]: write jobserver: Bad file descriptor. Stop. lto-wrapper: fatal error: make returned 2 exit status compilation terminated. /usr/bin/ld: error: lto-wrapper failed collect2: error: ld returned 1 exit status make[2]: [_deps/repo-core-build/python/tritonserver/CMakeFiles/python-bindings.dir/build.make:98: _deps/repo-core-build/python/tritonserver/triton_bindings.cpython-310-x86_64-linux-gnu.so] Error 1 make[2]: Leaving directory '/tmp/tritonbuild/tritonserver/build' make[1]: [CMakeFiles/Makefile2:757: _deps/repo-core-build/python/tritonserver/CMakeFiles/python-bindings.dir/all] Error 2 make[1]: Waiting for unfinished jobs.... -- Looking for dlfcn.h - found
What version of python are you using? Also, as a sanity check, is the version of "server" you are using the 23.10 release as well?
mujahiddh
commented
11 months ago I am using python 3.11.5 version, how do i confirm the server version, i am running the commands under the branch r23.10, i guess that would make the server version 23.10, right ?
mujahiddh
commented
11 months ago what is the right python version to use ?
fpetrini15
commented
11 months ago On closer inspection, I see that the output from build.py shows you're likely running the latest version of build.py from main:
Building Triton Inference Server platform linux machine x86_64 version 2.41.0dev build dir /..../server/build install dir None cmake dir None default repo-tag: main container version 23.12dev
When it should output the following for the 23.10 release:
Building Triton Inference Server platform linux machine x86_64 version 2.39.0 build dir /..../server/build install dir None cmake dir None default repo-tag: r23.10 container version 23.10
Can you try cloning the repo like this and building again: git clone -b r23.10 --single-branch https://github.com/triton-inference-server/server.git
mujahiddh
commented
11 months ago I just tried your suggestion and ended up with a similar error, this is the log at the end of the build
I would say the issue, still persist.
fpetrini15
commented
11 months ago What is the output of entering:
python3 build.py --dryrun
Also, to filter for the error in your build can you redirect your output to a file and find where it's failing:
./build.py --enable-logging .. {rest of your command} > output.txt
mujahiddh
commented
11 months ago This is the output Building Triton Inference Server platform linux machine x86_64 version 2.39.0 build dir /Users/m.abdullahi/Documents/GitHub/server/build install dir None cmake dir None default repo-tag: r23.10 container version 23.10 upstream container version 23.10 component "common" at tag/branch "r23.10" component "core" at tag/branch "r23.10" component "backend" at tag/branch "r23.10" component "thirdparty" at tag/branch "r23.10"
Yes, I have the logs in a file, just not sure what the issue is
mujahiddh
commented
11 months ago still stuck on this, will appreciate any help, will using an older release help ?
fpetrini15
commented
11 months ago Hi @mujahiddh,
Since the issue seems to be related to compiling the python bindings, you can try building the 23.08 release.
@GuanLuo are there any preconditions to compiling the python bindings / have you seen the above error before?
mujahiddh
commented
11 months ago I went back to 23.02 release, it seems to work, will check 23.08 at a later time. Thank you.
Description I get an error when trying to build custom triton image without, the error occurs in configuration step for grpc repo. I might be wrong but this is the only info i can see from the logs
Triton Information I am triton to build triton based on the 23.10 release
To Reproduce
output Building Triton Inference Server platform linux machine x86_64 version 2.41.0dev build dir /..../server/build install dir None cmake dir None default repo-tag: main container version 23.12dev upstream container version 23.10 endpoint "http" endpoint "grpc" filesystem "gcs" filesystem "s3" backend "custom" at tag/branch "main" backend "ensemble" at tag/branch "main" backend "onnxruntime" at tag/branch "r23.10" backend "tensorflow" at tag/branch "r23.10" backend "tensorflow" CMake extra "-DTRITON_TENSORFLOW_INSTALL_EXTRA_DEPS=ON" backend "tensorflow" CMake extra "-DTRITON_GRPC_INSTALL_EXTRA_DEPS=ON" component "common" at tag/branch "r23.10" component "core" at tag/branch "r23.10" component "backend" at tag/branch "r23.10" component "thirdparty" at tag/branch "r23.10" DEPRECATED: The legacy builder is deprecated and will be removed in a future release. Install the buildx component to build images with BuildKit: https://docs.docker.com/go/buildx/
Sending build context to Docker daemon 18.5MB Step 1/21 : ARG TRITON_VERSION=2.41.0dev Step 2/21 : ARG TRITON_CONTAINER_VERSION=23.12dev Step 3/21 : ARG BASE_IMAGE=ubuntu:22.04 Step 4/21 : FROM ${BASE_IMAGE} 22.04: Pulling from library/ubuntu Digest: sha256:8eab65df33a6de2844c9aefd19efe8ddb87b7df5e9185a4ab73af936225685bb Status: Image is up to date for ubuntu:22.04 ---> b6548eacb063 Step 5/21 : ARG TRITON_VERSION ---> Using cache ---> 5a2eaee21c7a Step 6/21 : ARG TRITON_CONTAINER_VERSION ---> Using cache ---> a6f890423ef7 Step 7/21 : ENV DEBIAN_FRONTEND=noninteractive ---> Using cache ---> a6e90ff93e0a Step 8/21 : RUN apt-get update && apt-get install -y ca-certificates curl gnupg && install -m 0755 -d /etc/apt/keyrings && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && chmod a+r /etc/apt/keyrings/docker.gpg && echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null && apt-get update && apt-get install -y docker.io docker-buildx-plugin ---> Using cache ---> 543728b9a28d Step 9/21 : RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates autoconf automake build-essential git gperf libre2-dev libssl-dev libtool libcurl4-openssl-dev libb64-dev libgoogle-perftools-dev patchelf python3-dev python3-pip python3-setuptools rapidjson-dev scons software-properties-common pkg-config unzip wget zlib1g-dev libarchive-dev libxml2-dev libnuma-dev wget && rm -rf /var/lib/apt/lists/ ---> Using cache ---> 5963d417c9d4 Step 10/21 : RUN pip3 install --upgrade pip && pip3 install --upgrade wheel setuptools docker ---> Using cache ---> 86a2482c2437 Step 11/21 : RUN wget -O /tmp/boost.tar.gz https://boostorg.jfrog.io/artifactory/main/release/1.80.0/source/boost_1_80_0.tar.gz && (cd /tmp && tar xzf boost.tar.gz) && cd /tmp/boost_1_80_0 && ./bootstrap.sh --prefix=/usr && ./b2 install && mv /tmp/boost_1_80_0/boost /usr/include/boost ---> Using cache ---> f4e202cfa04c Step 12/21 : RUN apt update && apt install -y gpg wget && wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null && . /etc/os-release && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null && apt-get update && apt-get install -y --no-install-recommends cmake cmake-data ---> Using cache ---> ad8d3ba9d1e9 Step 13/21 : ENV TRITON_SERVER_VERSION ${TRITON_VERSION} ---> Using cache ---> e832ea8d71d9 Step 14/21 : ENV NVIDIA_TRITON_SERVER_VERSION ${TRITON_CONTAINER_VERSION} ---> Using cache ---> dcf1b35c67bc Step 15/21 : WORKDIR /workspace ---> Using cache ---> e83dd40e5dd1 Step 16/21 : RUN rm -fr ---> Using cache ---> de4710fd6960 Step 17/21 : COPY . . ---> 03b6dc67abe3 Step 18/21 : ENTRYPOINT [] ---> Running in 3395404df105 Removing intermediate container 3395404df105 ---> 80b2b9b5c5c2 Step 19/21 : RUN mkdir -p /opt/ ---> Running in 32a39115b148 Removing intermediate container 32a39115b148 ---> 8fc77cd998d8 Step 20/21 : RUN wget "https://repo.anaconda.com/miniconda/Miniconda3-py310_23.1.0-1-Linux-x86_64.sh" -O miniconda.sh -q && echo "32d73e1bc33fda089d7cd9ef4c1be542616bd8e437d1f77afeeaf7afdb019787" "miniconda.sh" > shasum && sha256sum -c ./shasum && sh miniconda.sh -b -p /opt/conda && rm miniconda.sh shasum && find /opt/conda/ -follow -type f -name '.a' -delete && find /opt/conda/ -follow -type f -name '.js.map' -delete && /opt/conda/bin/conda clean -afy ---> Running in 8d296ae351bc miniconda.sh: OK PREFIX=/opt/conda Unpacking payload ... Extracting : python-3.10.9-h7a1cb2a_0.conda: 2%|▏ | 1/45 [00:00<00:33, 1Extracting : _libgcc_mutex-0.1-main.conda: 2%|▏ | 1/45 [00:00<00:33, 1.3Extracting : ca-certificates-2023.01.10-h06a4308_0.conda: 4%|▍ | 2/45 [00Extracting : ld_impl_linux-64-2.38-h1181459_1.conda: 7%|▋ | 3/45 [00:00<0Extracting : libstdcxx-ng-11.2.0-h1234567_1.conda: 9%|▉ | 4/45 [00:00<00:Extracting : tzdata-2022g-h04d1e81_0.conda: 11%|█ | 5/45 [00:00<00:30, 1.Extracting : libgomp-11.2.0-h1234567_1.conda: 13%|█▎ | 6/45 [00:00<00:29, Extracting : _openmp_mutex-5.1-1_gnu.conda: 16%|█▌ | 7/45 [00:00<00:28, 1.Extracting : libgcc-ng-11.2.0-h1234567_1.conda: 18%|█▊ | 8/45 [00:00<00:28,Extracting : bzip2-1.0.8-h7b6447c_0.conda: 20%|██ | 9/45 [00:00<00:27, 1.3Extracting : libffi-3.4.2-h6a678d5_6.conda: 22%|██▏ | 10/45 [00:00<00:26, 1Extracting : libuuid-1.41.5-h5eee18b_0.conda: 24%|██▍ | 11/45 [00:00<00:25, Extracting : ncurses-6.4-h6a678d5_0.conda: 27%|██▋ | 12/45 [00:00<00:25, 1.Extracting : openssl-1.1.1s-h7f8727e_0.conda: 29%|██▉ | 13/45 [00:00<00:24, Extracting : xz-5.2.10-h5eee18b_1.conda: 31%|███ | 14/45 [00:00<00:23, 1.32Extracting : zlib-1.2.13-h5eee18b_0.conda: 33%|███▎ | 15/45 [00:00<00:22, 1.Extracting : readline-8.2-h5eee18b_0.conda: 36%|███▌ | 16/45 [00:00<00:21, 1Extracting : tk-8.6.12-h1ccaba5_0.conda: 38%|███▊ | 17/45 [00:00<00:21, 1.32Extracting : sqlite-3.40.1-h5082296_0.conda: 40%|████ | 18/45 [00:00<00:20, Extracting : certifi-2022.12.7-py310h06a4308_0.conda: 42%|████▏ | 19/45 [00:00Extracting : charset-normalizer-2.0.4-pyhd3eb1b0_0.conda: 44%|████▍ | 20/45 [0Extracting : idna-3.4-py310h06a4308_0.conda: 47%|████▋ | 21/45 [00:00<00:18, Extracting : pluggy-1.0.0-py310h06a4308_1.conda: 49%|████▉ | 22/45 [00:00<00:1Extracting : pycosat-0.6.4-py310h5eee18b_0.conda: 51%|█████ | 23/45 [00:00<00:Extracting : pycparser-2.21-pyhd3eb1b0_0.conda: 53%|█████▎ | 24/45 [00:00<00:15Extracting : pysocks-1.7.1-py310h06a4308_0.conda: 56%|█████▌ | 25/45 [00:00<00:Extracting : ruamel.yaml.clib-0.2.6-py310h5eee18b_1.conda: 58%|█████▊ | 26/45 [Extracting : six-1.16.0-pyhd3eb1b0_1.conda: 60%|██████ | 27/45 [00:00<00:13, 1Extracting : toolz-0.12.0-py310h06a4308_0.conda: 62%|██████▏ | 28/45 [00:00<00:1Extracting : tqdm-4.64.1-py310h06a4308_0.conda: 64%|██████▍ | 29/45 [00:00<00:12Extracting : wheel-0.37.1-pyhd3eb1b0_0.conda: 67%|██████▋ | 30/45 [00:00<00:11, Extracting : cffi-1.15.1-py310h5eee18b_3.conda: 69%|██████▉ | 31/45 [00:00<00:10Extracting : ruamel.yaml-0.17.21-py310h5eee18b_0.conda: 71%|███████ | 32/45 [00:Extracting : setuptools-65.6.3-py310h06a4308_0.conda: 73%|███████▎ | 33/45 [00:00Extracting : brotlipy-0.7.0-py310h7f8727e_1002.conda: 76%|███████▌ | 34/45 [00:00Extracting : cryptography-38.0.4-py310h9ce1e76_0.conda: 78%|███████▊ | 35/45 [00:Extracting : pip-22.3.1-py310h06a4308_0.conda: 80%|████████ | 36/45 [00:00<00:06,Extracting : zstandard-0.18.0-py310h5eee18b_0.conda: 82%|████████▏ | 37/45 [00:00<Extracting : conda-content-trust-0.1.3-py310h06a4308_0.conda: 84%|████████▍ | 38/4Extracting : conda-package-streaming-0.7.0-py310h06a4308_0.conda: 87%|████████▋ | Extracting : pyopenssl-22.0.0-pyhd3eb1b0_0.conda: 89%|████████▉ | 40/45 [00:00<00:Extracting : conda-package-handling-2.0.2-py310h06a4308_0.conda: 91%|█████████ | 4Extracting : urllib3-1.26.14-py310h06a4308_0.conda: 93%|█████████▎| 42/45 [00:00<0Extracting : requests-2.28.1-py310h06a4308_0.conda: 96%|█████████▌| 43/45 [00:00<0Extracting : conda-23.1.0-py310h06a4308_0.conda: 98%|█████████▊| 44/45 [00:00<00:0 Installing base environment...