replicate / cog

Containers for machine learning
https://cog.run
Apache License 2.0
8.1k stars 564 forks source link

how to add dlib? #2037

Closed AlexanderKozhevin closed 3 weeks ago

AlexanderKozhevin commented 3 weeks ago

getting error everytime

40.46         File "/tmp/pip-install-do0sqbbq/dlib_4d1149fb1c7d4b0fbcc927d6d4462767/setup.py", line 171, in build_extension
40.46           subprocess.check_call(cmake_setup, cwd=build_folder)
40.46         File "/root/.pyenv/versions/3.11.10/lib/python3.11/subprocess.py", line 413, in check_call
40.46           raise CalledProcessError(retcode, cmd)
40.46       subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-install-do0sqbbq/dlib_4d1149fb1c7d4b0fbcc927d6d4462767/tools/python', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-install-do0sqbbq/dlib_4d1149fb1c7d4b0fbcc927d6d4462767/build/lib.linux-x86_64-cpython-311', '-DPYTHON_EXECUTABLE=/root/.pyenv/versions/3.11.10/bin/python3.11', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
40.46       [end of output]
40.46   
40.46   note: This error originates from a subprocess, and is likely not a problem with pip.
40.46   ERROR: Failed building wheel for dlib
40.46   Running setup.py clean for dlib
40.55 Failed to build dlib
40.55 ERROR: Could not build wheels for dlib, which is required to install pyproject.toml-based projects
40.75 
40.75 [notice] A new release of pip is available: 24.0 -> 24.3.1
40.75 [notice] To update, run: pip install --upgrade pip
------
Dockerfile:10
--------------------
   8 |     COPY .cog/tmp/build20241030183754.2830722455133927/requirements.txt /tmp/requirements.txt
   9 |     ENV CFLAGS="-O3 -funroll-loops -fno-strict-aliasing -flto -S"
  10 | >>> RUN --mount=type=cache,target=/root/.cache/pip pip install -r /tmp/requirements.txt
  11 |     ENV CFLAGS=
  12 |     RUN unzip downloads/ninja-linux.zip -d /usr/local/bin/
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -r /tmp/requirements.txt" did not complete successfully: exit code: 1
ⅹ Failed to build Docker image: exit status 1
build:
  gpu: true
  cuda: '12.1'
  system_packages:
    - "unzip"
    - "libx11-dev"
    - "libopenblas-dev"
    - "liblapack-dev"
    - "build-essential"
    - "python3-dev"
    - "pkg-config"
    - "software-properties-common"
  pre_install:
    - "apt-get update"
    - "apt-get install -y software-properties-common"
    - "add-apt-repository -y ppa:ubuntu-toolchain-r/test"
    - "add-apt-repository -y ppa:kitware/cmake-3.x"
    - "apt-get update"
    - "apt-get install -y gcc-11 g++-11"
    - "update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 60 --slave /usr/bin/g++ g++ /usr/bin/g++-11"
    - "apt-get install -y cmake"
  python_version: "3.11"  # Specifically using Python 3.8
  python_packages:
    - "numpy"
    - "torch==2.1.2+cu121"
    - "torchvision==0.16.2+cu121"
    - "Pillow==9.5.0"
    - "dlib==19.24.0"
    - "opencv-python"
    - "scipy"
    - "scikit-image"
    - "matplotlib"
    - "huggingface_hub"
    - "requests"
    - "tqdm"
  run:
    - "unzip downloads/ninja-linux.zip -d /usr/local/bin/"
    - "update-alternatives --install /usr/bin/ninja ninja /usr/local/bin/ninja 1 --force"

predict: "predict.py:Predictor"
AlexanderKozhevin commented 3 weeks ago

cog debug

sudo cog debug
[sudo] password for alexpolymath: 
Sorry, try again.
[sudo] password for alexpolymath: 
#syntax=docker/dockerfile:1.4
FROM r8.im/cog-base:cuda12.1-python3.11
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked apt-get update -qq && apt-get install -qqy libboost-all-dev && rm -rf /var/lib/apt/lists/*
COPY .cog/tmp/build20241030235441.4814153613635754/cog-0.11.6-py3-none-any.whl /tmp/cog-0.11.6-py3-none-any.whl
ENV CFLAGS="-O3 -funroll-loops -fno-strict-aliasing -flto -S"
RUN --mount=type=cache,target=/root/.cache/pip pip install --no-cache-dir /tmp/cog-0.11.6-py3-none-any.whl 'pydantic<2'
ENV CFLAGS=
COPY .cog/tmp/build20241030235441.4814153613635754/requirements.txt /tmp/requirements.txt
ENV CFLAGS="-O3 -funroll-loops -fno-strict-aliasing -flto -S"
RUN --mount=type=cache,target=/root/.cache/pip pip install -r /tmp/requirements.txt
ENV CFLAGS=
RUN apt-get update && apt-get install -y cmake
WORKDIR /src
EXPOSE 5000
CMD ["python", "-m", "cog.server.http"]
COPY . /src

i runs cmake install after python deps. That's basically wrong

AlexanderKozhevin commented 3 weeks ago

here is a hack which worked for me

build:
  gpu: true
  cuda: "11.8"
  python_version: "3.11"
  system_packages:
    - "vim"
    - "ffmpeg"
    - "libsm6"
    - "libxext6"
    - "cmake"
    - "build-essential"
    - "libboost-all-dev"
    - "libopenblas-dev"    # Add this for linear algebra dependencies
    - "liblapack-dev"      # Add this for additional math support
    - "pkg-config"         # Necessary for some package builds
    - "libx11-dev"
    - "libgtk-3-dev"
    - "libboost-all-dev"
  python_packages:
    - "setuptools==71.0.0"
    - "distlib==0.3.7"
    - "cmake==3.27.5"
  pre_install:
    - "pip install dlib"
predict: "predict.py:Predictor"