search

dusty-nv / jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
MIT License
1.93k stars 422 forks source link

Not able to build mlc #404

Open raj-khare opened 4 months ago

raj-khare commented 4 months ago

Steps to reproduce:

  1. git clone https://github.com/dusty-nv/jetson-containers
  2. cd jetson-containers
  3. ./build.sh mlc:dev --skip-tests all

Error:

truffle@tegra-ubuntu:~/jetson-containers$ ./build.sh mlc:dev --skip-tests all
Namespace(packages=['mlc:dev'], name='', base='', multiple=False, build_flags='', package_dirs=[''], list_packages=False, show_packages=False, skip_packages=[''], skip_errors=False, skip_tests=['all'], test_only=[''], simulate=False, push='', logs='', verbose=False, no_github_api=False)
-- L4T_VERSION=36.2.0
-- JETPACK_VERSION=6.0
-- CUDA_VERSION=12.2.140
-- LSB_RELEASE=22.04 (jammy)
-- Package text-generation-inference has missing dependencies, disabling...  ("couldn't find package:  bitsandbytes")
-- Package l4t-text-generation has missing dependencies, disabling...  ("couldn't find package:  bitsandbytes")
-- Package l4t-tensorflow:tf1 has missing dependencies, disabling...  ("couldn't find package:  tensorflow")
-- Building containers  ['build-essential', 'cuda', 'cudnn', 'python', 'tensorrt', 'numpy', 'cmake', 'onnx', 'pytorch', 'torchvision', 'huggingface_hub', 'rust', 'transformers', 'mlc:dev']
-- Building container mlc:dev-r36.2.0-build-essential

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-build-essential \
--file /home/truffle/jetson-containers/packages/build-essential/Dockerfile \
--build-arg BASE_IMAGE=ubuntu:22.04 \
/home/truffle/jetson-containers/packages/build-essential \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-build-essential.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  13.82kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 9cbdd1f76112
Step 3/5 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 9127a1e709f4
Step 4/5 : RUN apt-get update &&     apt-get install -y --no-install-recommends                   build-essential                 software-properties-common   apt-transport-https             ca-certificates                 lsb-release             pkg-config              gnupg                   git             wget                 curl            nano            zip             unzip     && rm -rf /var/lib/apt/lists/*     && apt-get clean
 ---> Using cache
 ---> a8001bbc7af1
Step 5/5 : RUN gcc --version &&     g++ --version
 ---> Using cache
 ---> 490481ba1c70
Successfully built 490481ba1c70
Successfully tagged mlc:dev-r36.2.0-build-essential
-- Building container mlc:dev-r36.2.0-cuda

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-cuda \
--file /home/truffle/jetson-containers/packages/cuda/cuda/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-build-essential \
--build-arg CUDA_URL="https://nvidia.box.com/shared/static/uvqtun1sc0bq76egarc8wwuh6c23e76e.deb" \
--build-arg CUDA_DEB="cuda-tegra-repo-ubuntu2204-12-2-local" \
--build-arg CUDA_PACKAGES="cuda-toolkit*" \
/home/truffle/jetson-containers/packages/cuda/cuda \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-cuda.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  26.11kB
Step 1/14 : ARG BASE_IMAGE
Step 2/14 : FROM ${BASE_IMAGE}
 ---> 490481ba1c70
Step 3/14 : ARG CUDA_URL
 ---> Using cache
 ---> dc0bd1caaf24
Step 4/14 : ARG CUDA_DEB
 ---> Using cache
 ---> c085fbb7b158
Step 5/14 : ARG CUDA_PACKAGES
 ---> Using cache
 ---> f84d5f637a3f
Step 6/14 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> ffa15b780cb8
Step 7/14 : RUN apt-get update &&     apt-get install -y --no-install-recommends             wget                 git             binutils                xz-utils             ca-certificates     && rm -rf /var/lib/apt/lists/*     && apt-get clean
 ---> Using cache
 ---> d134aca3665f
Step 8/14 : RUN echo "Downloading ${CUDA_DEB}" &&     mkdir /tmp/cuda && cd /tmp/cuda &&     wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/arm64/cuda-ubuntu2004.pin -O /etc/apt/preferences.d/cuda-repository-pin-600 &&     wget --quiet --show-progress --progress=bar:force:noscroll ${CUDA_URL} &&     dpkg -i *.deb &&     cp /var/cuda-tegra-repo-*/cuda-tegra-*-keyring.gpg /usr/share/keyrings/ &&     ar x /var/cuda-tegra-repo-*/cuda-compat-*.deb &&     tar xvf data.tar.xz -C / &&     apt-get update &&     apt-get install -y --no-install-recommends ${CUDA_PACKAGES} &&     rm -rf /var/lib/apt/lists/* &&     apt-get clean &&     dpkg --list | grep cuda &&     dpkg -P ${CUDA_DEB} &&     rm -rf /tmp/cuda
 ---> Using cache
 ---> 8ab24af0d9c6
Step 9/14 : ENV NVIDIA_VISIBLE_DEVICES=all
 ---> Using cache
 ---> f18d3b645608
Step 10/14 : ENV NVIDIA_DRIVER_CAPABILITIES=all
 ---> Using cache
 ---> b2d36078e23d
Step 11/14 : ENV CUDA_HOME="/usr/local/cuda"
 ---> Using cache
 ---> f3f0c356b87b
Step 12/14 : ENV PATH="/usr/local/cuda/bin:${PATH}"
 ---> Using cache
 ---> 50bc1984cc86
Step 13/14 : ENV LD_LIBRARY_PATH="/usr/local/cuda/compat:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
 ---> Using cache
 ---> 8f5d6f9ea3d9
Step 14/14 : WORKDIR /
 ---> Using cache
 ---> e73593c11778
Successfully built e73593c11778
Successfully tagged mlc:dev-r36.2.0-cuda
-- Building container mlc:dev-r36.2.0-cudnn

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-cudnn \
--file /home/truffle/jetson-containers/packages/cuda/cudnn/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-cuda \
--build-arg CUDNN_URL="https://nvidia.box.com/shared/static/ht4li6b0j365ta7b76a6gw29rk5xh8cy.deb" \
--build-arg CUDNN_DEB="cudnn-local-tegra-repo-ubuntu2204-8.9.4.25" \
--build-arg CUDNN_PACKAGES="libcudnn*-dev libcudnn*-samples" \
/home/truffle/jetson-containers/packages/cuda/cudnn \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-cudnn.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  22.02kB
Step 1/7 : ARG BASE_IMAGE
Step 2/7 : FROM ${BASE_IMAGE}
 ---> e73593c11778
Step 3/7 : ARG CUDNN_URL
 ---> Using cache
 ---> b57eaedb35a2
Step 4/7 : ARG CUDNN_DEB
 ---> Using cache
 ---> 2212f6c88ef6
Step 5/7 : ARG CUDNN_PACKAGES
 ---> Using cache
 ---> d6e6691badd1
Step 6/7 : RUN echo "Downloading ${CUDNN_DEB}" &&     mkdir /tmp/cudnn && cd /tmp/cudnn &&     wget --quiet --show-progress --progress=bar:force:noscroll ${CUDNN_URL} &&     dpkg -i *.deb &&     cp /var/cudnn-local-tegra-repo-*/cudnn-local-tegra-*-keyring.gpg /usr/share/keyrings/ &&     apt-get update &&     apt-cache search cudnn &&     apt-get install -y --no-install-recommends ${CUDNN_PACKAGES} &&     rm -rf /var/lib/apt/lists/* &&     apt-get clean &&     dpkg --list | grep cudnn &&     dpkg -P ${CUDNN_DEB} &&     rm -rf /tmp/cudnn
 ---> Using cache
 ---> aaea91ff9742
Step 7/7 : RUN cd /usr/src/cudnn_samples_v8/conv_sample/ &&     make -j$(nproc)
 ---> Using cache
 ---> c96cdcd10d45
Successfully built c96cdcd10d45
Successfully tagged mlc:dev-r36.2.0-cudnn
-- Building container mlc:dev-r36.2.0-python

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-python \
--file /home/truffle/jetson-containers/packages/python/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-cudnn \
/home/truffle/jetson-containers/packages/python \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-python.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  13.82kB
Step 1/8 : ARG BASE_IMAGE
Step 2/8 : FROM ${BASE_IMAGE}
 ---> c96cdcd10d45
Step 3/8 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 5a34795c7c13
Step 4/8 : RUN apt-get update &&     apt-get install -y --no-install-recommends                   python3-dev             python3-pip     && rm -rf /var/lib/apt/lists/*     && apt-get clean
 ---> Using cache
 ---> 46185cce4f54
Step 5/8 : RUN which pip3 && pip3 --version &&     python3 -m pip install --upgrade pip
 ---> Using cache
 ---> 7c563d044b26
Step 6/8 : RUN which python || python --version || pip --version ||     which python3 || python3 --version || pip3 --version
 ---> Using cache
 ---> fb5f66947a17
Step 7/8 : RUN pip3 install --upgrade --no-cache-dir    setuptools      packaging       'Cython<3'      wheel
 ---> Using cache
 ---> b740ec40f5a3
Step 8/8 : RUN pip3 install --no-cache-dir --verbose wget psutil
 ---> Using cache
 ---> f57eb9148c73
Successfully built f57eb9148c73
Successfully tagged mlc:dev-r36.2.0-python
-- Building container mlc:dev-r36.2.0-tensorrt

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-tensorrt \
--file /home/truffle/jetson-containers/packages/tensorrt/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-python \
--build-arg TENSORRT_URL="https://nvidia.box.com/shared/static/hmwr57hm88bxqrycvlyma34c3k4c53t9.deb" \
--build-arg TENSORRT_DEB="nv-tensorrt-local-repo-l4t-8.6.2-cuda-12.2" \
--build-arg TENSORRT_PACKAGES="tensorrt tensorrt-libs python3-libnvinfer-dev" \
/home/truffle/jetson-containers/packages/tensorrt \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-tensorrt.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon   21.5kB
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM ${BASE_IMAGE}
 ---> f57eb9148c73
Step 3/6 : ARG TENSORRT_URL
 ---> Using cache
 ---> 7ce18e5f6cb2
Step 4/6 : ARG TENSORRT_DEB
 ---> Using cache
 ---> 2f94ef535108
Step 5/6 : ARG TENSORRT_PACKAGES
 ---> Using cache
 ---> 0dfa4b541e22
Step 6/6 : RUN echo "Downloading ${TENSORRT_DEB}" &&     mkdir /tmp/cudnn && cd /tmp/cudnn &&     wget --quiet --show-progress --progress=bar:force:noscroll ${TENSORRT_URL} &&     dpkg -i *.deb &&     cp /var/nv-tensorrt-local-repo-*/nv-tensorrt-local-*-keyring.gpg /usr/share/keyrings/ &&     apt-get update &&     apt-cache search tensorrt &&     apt-get install -y --no-install-recommends ${TENSORRT_PACKAGES} &&     rm -rf /var/lib/apt/lists/* &&     apt-get clean &&     dpkg --list | grep tensorrt &&     dpkg -P ${TENSORRT_DEB} &&     rm -rf /tmp/tensorrt
 ---> Using cache
 ---> 92a68ff3f037
Successfully built 92a68ff3f037
Successfully tagged mlc:dev-r36.2.0-tensorrt
-- Building container mlc:dev-r36.2.0-numpy

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-numpy \
--file /home/truffle/jetson-containers/packages/numpy/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-tensorrt \
/home/truffle/jetson-containers/packages/numpy \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-numpy.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  13.82kB
Step 1/7 : ARG BASE_IMAGE
Step 2/7 : FROM ${BASE_IMAGE}
 ---> 92a68ff3f037
Step 3/7 : RUN apt-get update &&     apt-get install -y --no-install-recommends                   python3-numpy     && rm -rf /var/lib/apt/lists/*     && apt-get clean
 ---> Using cache
 ---> e2b87660c4e8
Step 4/7 : ENV OPENBLAS_CORETYPE=ARMV8
 ---> Using cache
 ---> b2c361a7fdab
Step 5/7 : RUN pip3 show numpy && python3 -c 'import numpy; print(numpy.__version__)'
 ---> Using cache
 ---> 14d894d232ef
Step 6/7 : RUN pip3 install --upgrade --no-cache-dir --verbose numpy
 ---> Using cache
 ---> 8368a25bb84c
Step 7/7 : RUN pip3 show numpy && python3 -c 'import numpy; print(numpy.__version__)'
 ---> Using cache
 ---> b92d3c94a38a
Successfully built b92d3c94a38a
Successfully tagged mlc:dev-r36.2.0-numpy
-- Building container mlc:dev-r36.2.0-cmake

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-cmake \
--file /home/truffle/jetson-containers/packages/cmake/cmake_pip/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-numpy \
/home/truffle/jetson-containers/packages/cmake/cmake_pip \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-cmake.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  10.75kB
Step 1/4 : ARG BASE_IMAGE
Step 2/4 : FROM ${BASE_IMAGE}
 ---> b92d3c94a38a
Step 3/4 : RUN pip3 install --upgrade --force-reinstall --no-cache-dir --verbose cmake
 ---> Using cache
 ---> e9de8b6523b4
Step 4/4 : RUN cmake --version && which cmake
 ---> Using cache
 ---> 0e6faa88be7b
Successfully built 0e6faa88be7b
Successfully tagged mlc:dev-r36.2.0-cmake
-- Building container mlc:dev-r36.2.0-onnx

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-onnx \
--file /home/truffle/jetson-containers/packages/onnx/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-cmake \
--build-arg ONNX_VERSION="main" \
/home/truffle/jetson-containers/packages/onnx \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-onnx.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  14.85kB
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM ${BASE_IMAGE}
 ---> 0e6faa88be7b
Step 3/6 : ARG ONNX_VERSION
 ---> Using cache
 ---> 8cdd0e8e716a
Step 4/6 : RUN if [ `lsb_release --codename --short` != 'bionic' ]; then       pip3 install --no-cache-dir --verbose protobuf==4.22.3;     fi
 ---> Using cache
 ---> 077a5ace3897
Step 5/6 : RUN pip3 install --no-cache-dir --verbose git+https://github.com/onnx/onnx@${ONNX_VERSION}
 ---> Using cache
 ---> b0fc116d8494
Step 6/6 : RUN pip3 show onnx && python3 -c 'import onnx; print(onnx.__version__)'
 ---> Using cache
 ---> 4fb831665373
Successfully built 4fb831665373
Successfully tagged mlc:dev-r36.2.0-onnx
-- Building container mlc:dev-r36.2.0-pytorch

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-pytorch \
--file /home/truffle/jetson-containers/packages/pytorch/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-onnx \
--build-arg PYTORCH_WHL="torch-2.1.0-cp310-cp310-linux_aarch64.whl" \
--build-arg PYTORCH_URL="https://nvidia.box.com/shared/static/0h6tk4msrl9xz3evft9t0mpwwwkw7a32.whl" \
--build-arg TORCH_CUDA_ARCH_ARGS="8.7" \
/home/truffle/jetson-containers/packages/pytorch \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-pytorch.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon    105kB
Step 1/14 : ARG BASE_IMAGE
Step 2/14 : FROM ${BASE_IMAGE}
 ---> 4fb831665373
Step 3/14 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 6f4a5fb1f11e
Step 4/14 : RUN apt-get update &&     apt-get install -y --no-install-recommends                  libopenblas-dev                 libopenmpi-dev             openmpi-bin             openmpi-common            gfortran                libomp-dev     && rm -rf /var/lib/apt/lists/*     && apt-get clean
 ---> Using cache
 ---> 7acdce33b08f
Step 5/14 : ARG PYTORCH_URL
 ---> Using cache
 ---> a1322a4435d7
Step 6/14 : ARG PYTORCH_WHL
 ---> Using cache
 ---> 0a4a9086c0fc
Step 7/14 : RUN cd /opt &&     wget --quiet --show-progress --progress=bar:force:noscroll --no-check-certificate ${PYTORCH_URL} -O ${PYTORCH_WHL} &&     pip3 install --verbose ${PYTORCH_WHL}
 ---> Using cache
 ---> 3b628e7374c2
Step 8/14 : RUN python3 -c 'import torch; print(f"PyTorch version: {torch.__version__}"); print(f"CUDA available:  {torch.cuda.is_available()}"); print(f"cuDNN version:   {torch.backends.cudnn.version()}"); print(torch.__config__.show());'
 ---> Using cache
 ---> 086f145ae860
Step 9/14 : RUN PYTHON_ROOT=`pip3 show torch | grep Location: | cut -d' ' -f2` &&     TORCH_CMAKE_CONFIG=$PYTHON_ROOT/torch/share/cmake/Torch/TorchConfig.cmake &&     echo "patching _GLIBCXX_USE_CXX11_ABI in ${TORCH_CMAKE_CONFIG}" &&     sed -i 's/  set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=")/  set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")/g' ${TORCH_CMAKE_CONFIG}
 ---> Using cache
 ---> cd9ce20c7a82
Step 10/14 : RUN pip3 install --no-cache-dir scikit-build &&     pip3 install --no-cache-dir ninja
 ---> Using cache
 ---> 967a06d9314d
Step 11/14 : ARG TORCH_CUDA_ARCH_ARGS
 ---> Using cache
 ---> d385f4c2715a
Step 12/14 : ENV TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_ARGS}
 ---> Using cache
 ---> 8d93851e872c
Step 13/14 : RUN echo "TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_LIST}"
 ---> Using cache
 ---> 17a8a414d68e
Step 14/14 : ENV TORCH_HOME=/data/models/torch
 ---> Using cache
 ---> 98ac12aec176
Successfully built 98ac12aec176
Successfully tagged mlc:dev-r36.2.0-pytorch
-- Building container mlc:dev-r36.2.0-torchvision

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-torchvision \
--file /home/truffle/jetson-containers/packages/pytorch/torchvision/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-pytorch \
--build-arg TORCHVISION_VERSION="v0.16.0" \
--build-arg TORCH_CUDA_ARCH_LIST="8.7" \
/home/truffle/jetson-containers/packages/pytorch/torchvision \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-torchvision.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  24.58kB
Step 1/9 : ARG BASE_IMAGE
Step 2/9 : FROM ${BASE_IMAGE}
 ---> 98ac12aec176
Step 3/9 : ARG TORCHVISION_VERSION
 ---> Using cache
 ---> 330cbd2138c7
Step 4/9 : ARG TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2;8.7"
 ---> Using cache
 ---> 60088058b6a8
Step 5/9 : RUN printenv && echo "torchvision version = $TORCHVISION_VERSION" && echo "TORCH_CUDA_ARCH_LIST = $TORCH_CUDA_ARCH_LIST"
 ---> Using cache
 ---> 9a95298425fb
Step 6/9 : RUN apt-get update &&     apt-get install -y --no-install-recommends             libjpeg-dev                   zlib1g-dev     && rm -rf /var/lib/apt/lists/*     && apt-get clean
 ---> Using cache
 ---> 1071626e8580
Step 7/9 : RUN git clone --branch ${TORCHVISION_VERSION} --recursive --depth=1 https://github.com/pytorch/vision torchvision &&     cd torchvision &&     git checkout ${TORCHVISION_VERSION} &&     python3 setup.py bdist_wheel &&     cp dist/torchvision*.whl /opt &&     rm -rf ../torchvision
 ---> Using cache
 ---> 8a6a301f5add
Step 8/9 : RUN pip3 install --no-cache-dir --verbose /opt/torchvision*.whl
 ---> Using cache
 ---> be91312ad553
Step 9/9 : RUN python3 -c 'import torchvision; print(torchvision.__version__);'
 ---> Using cache
 ---> 8537e4c40ab1
Successfully built 8537e4c40ab1
Successfully tagged mlc:dev-r36.2.0-torchvision
-- Building container mlc:dev-r36.2.0-huggingface_hub

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-huggingface_hub \
--file /home/truffle/jetson-containers/packages/llm/huggingface_hub/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-torchvision \
/home/truffle/jetson-containers/packages/llm/huggingface_hub \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-huggingface_hub.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon   16.9kB
Step 1/11 : ARG BASE_IMAGE
Step 2/11 : FROM ${BASE_IMAGE}
 ---> 8537e4c40ab1
Step 3/11 : RUN pip3 install --no-cache-dir --verbose huggingface_hub[cli]
 ---> Using cache
 ---> b772030a8a3d
Step 4/11 : RUN pip3 install --no-cache-dir --verbose dataclasses
 ---> Using cache
 ---> 9c6247cbd27d
Step 5/11 : ENV TRANSFORMERS_CACHE=/data/models/huggingface
 ---> Using cache
 ---> ea5e9087c95f
Step 6/11 : ENV HUGGINGFACE_HUB_CACHE=/data/models/huggingface
 ---> Using cache
 ---> de3bf1e0d2ea
Step 7/11 : ENV HF_HOME=/data/models/huggingface
 ---> Using cache
 ---> e5164c137771
Step 8/11 : COPY huggingface-downloader /usr/local/bin/
 ---> Using cache
 ---> 078dc6bbbc12
Step 9/11 : COPY huggingface-downloader.py /usr/local/bin/_huggingface-downloader.py
 ---> Using cache
 ---> 77996d0d78b1
Step 10/11 : RUN huggingface-cli --help &&     huggingface-downloader --help &&     pip3 show huggingface_hub &&     python3 -c 'import huggingface_hub; print(huggingface_hub.__version__)'
 ---> Using cache
 ---> ef4b979aad67
Step 11/11 : RUN apt-get update &&     apt-get install -y --no-install-recommends time     && rm -rf /var/lib/apt/lists/*     && apt-get clean
 ---> Using cache
 ---> 25286343cef7
Successfully built 25286343cef7
Successfully tagged mlc:dev-r36.2.0-huggingface_hub
-- Building container mlc:dev-r36.2.0-rust

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-rust \
--file /home/truffle/jetson-containers/packages/rust/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-huggingface_hub \
/home/truffle/jetson-containers/packages/rust \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-rust.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  9.216kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> 25286343cef7
Step 3/5 : RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
 ---> Using cache
 ---> 85608e214cee
Step 4/5 : ENV PATH="/root/.cargo/bin:${PATH}"
 ---> Using cache
 ---> 6e3c0e350960
Step 5/5 : RUN rustc --version &&     pip3 install --no-cache-dir --verbose setuptools-rust
 ---> Using cache
 ---> f6037e255e5b
Successfully built f6037e255e5b
Successfully tagged mlc:dev-r36.2.0-rust
-- Building container mlc:dev-r36.2.0-transformers

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-transformers \
--file /home/truffle/jetson-containers/packages/llm/transformers/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-rust \
/home/truffle/jetson-containers/packages/llm/transformers \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-transformers.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  32.26kB
Step 1/10 : ARG BASE_IMAGE
Step 2/10 : FROM ${BASE_IMAGE}
 ---> f6037e255e5b
Step 3/10 : ARG TRANSFORMERS_PACKAGE=transformers
 ---> Using cache
 ---> 3a4ce9f336b7
Step 4/10 : ARG TRANSFORMERS_VERSION=https://pypi.org/pypi/transformers/json
 ---> Using cache
 ---> ab2f8e49aff2
Step 5/10 : ADD ${TRANSFORMERS_VERSION} /tmp/transformers_version.json

 ---> Using cache
 ---> ade7f822a71b
Step 6/10 : RUN pip3 install --no-cache-dir --verbose   accelerate      optimum         sentencepiece
 ---> Using cache
 ---> 346d3dd289ef
Step 7/10 : RUN pip3 uninstall -y transformers &&     pip3 install --no-cache-dir --verbose ${TRANSFORMERS_PACKAGE}
 ---> Using cache
 ---> 5cb35f9fe043
Step 8/10 : RUN PYTHON_ROOT=`pip3 show transformers | grep Location: | cut -d' ' -f2` &&     sed -i 's|torch.distributed.is_initialized|torch.distributed.is_available|g' -i ${PYTHON_ROOT}/transformers/modeling_utils.py
 ---> Using cache
 ---> 25f418f04d32
Step 9/10 : COPY huggingface-benchmark.py /usr/local/bin
 ---> Using cache
 ---> 7a4641286f18
Step 10/10 : RUN pip3 show transformers && python3 -c 'import transformers; print(transformers.__version__)'
 ---> Using cache
 ---> b63e0fc74c7b
Successfully built b63e0fc74c7b
Successfully tagged mlc:dev-r36.2.0-transformers
-- Building container mlc:dev-r36.2.0-mlc_dev

DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-mlc_dev \
--file /home/truffle/jetson-containers/packages/llm/mlc/Dockerfile \
--build-arg BASE_IMAGE=mlc:dev-r36.2.0-transformers \
--build-arg BUILD_IMAGE="None" \
--build-arg PYTHON_VERSION="3.10" \
--build-arg MLC_REPO="mlc-ai/mlc-llm" \
--build-arg MLC_VERSION="1cbd67b4eefe8f9cd8f4a1d798221483ec120fe9" \
--build-arg MLC_PATCH="patches/3feed05.diff" \
/home/truffle/jetson-containers/packages/llm/mlc \
2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-mlc_dev.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  112.1kB
Step 1/23 : ARG BASE_IMAGE
Step 2/23 : ARG BUILD_IMAGE
Step 3/23 : FROM ${BUILD_IMAGE} as builder
invalid reference format: repository name (library/None) must be lowercase
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/truffle/jetson-containers/jetson_containers/build.py", line 102, in <module>
    build_container(args.name, args.packages, args.base, args.build_flags, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api)
  File "/home/truffle/jetson-containers/jetson_containers/container.py", line 142, in build_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)  
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag mlc:dev-r36.2.0-mlc_dev --file /home/truffle/jetson-containers/packages/llm/mlc/Dockerfile --build-arg BASE_IMAGE=mlc:dev-r36.2.0-transformers --build-arg BUILD_IMAGE="None" --build-arg PYTHON_VERSION="3.10" --build-arg MLC_REPO="mlc-ai/mlc-llm" --build-arg MLC_VERSION="1cbd67b4eefe8f9cd8f4a1d798221483ec120fe9" --build-arg MLC_PATCH="patches/3feed05.diff" /home/truffle/jetson-containers/packages/llm/mlc 2>&1 | tee /home/truffle/jetson-containers/logs/20240226_231551/build/mlc_dev-r36.2.0-mlc_dev.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.
raj-khare commented 4 months ago

BUILD_IMAGE is not getting set properly

dusty-nv commented 4 months ago

@raj-khare try running ./build.sh mlc:dev-builder first. I think if you would have just done ./build.sh mlc it would have worked because those builder images are already populated on dockerhub (for example dustynv/mlc:51fb0f4-builder-r36.2.0)