[Bug]: Python SDK installation fails inside docker with image python:3.10-alpine

[onel]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.3
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: 
- GPU:
- Others:

Current Behavior

Adding pymilvus to requirements and then trying to build the docker image fails because of pyarrow. Error:

Building wheels for collected packages: pyarrow
  Building wheel for pyarrow (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyarrow (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [297 lines of output]
      <string>:34: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
      Traceback (most recent call last):
        File "/tmp/pip-build-env-1gj1kt4w/overlay/lib/python3.10/site-packages/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject
          section = defn.get("tool", {})[tool_name]
      KeyError: 'setuptools_scm'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/pyarrow
      copying pyarrow/fs.py -> build/lib.linux-x86_64-cpython-310/pyarrow
      ...
      warning: build_py: byte-compiling is disabled, skipping.

      running build_ext
      creating /tmp/pip-install-mn0dnz95/pyarrow_334d693220ee4c25826fa2758c0e8ab4/build/temp.linux-x86_64-cpython-310
      -- Running cmake for PyArrow
      cmake -DCMAKE_INSTALL_PREFIX=/tmp/pip-install-mn0dnz95/pyarrow_334d693220ee4c25826fa2758c0e8ab4/build/lib.linux-x86_64-cpython-310/pyarrow -DPYTHON_EXECUTABLE=/usr/local/bin/python -DPython3_EXECUTABLE=/usr/local/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_AZURE=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /tmp/pip-install-mn0dnz95/pyarrow_334d693220ee4c25826fa2758c0e8ab4
      error: command 'cmake' failed: No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects

Expected Behavior

SDK install should work

Steps To Reproduce

1. add a requirements.txt with pymilvus==2.4.3
2. run `docker compose buid`
3. build fails because of error

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @XuanYang-cn /unassign

[XuanYang-cn]

@onel It seems it failed when compiling PyArrow. But pymilvus has removed PyArrow as the dependencies in 2.4.3 https://github.com/milvus-io/pymilvus/pull/2087

Can you provide more information on how to reproducing this problem? /assign @onel

[onel]

Sure thing. This is how I recreated it.

In a folder add these 2 files: Dockerfile

# base image
FROM python:3.12-alpine

# setup environment variable
ENV DockerHOME=/app/

# set work directory
RUN mkdir -p $DockerHOME

# where your code lives
WORKDIR $DockerHOME

RUN pip install --no-cache --upgrade pip setuptools

# copy whole project to your docker home directory. 
COPY . $DockerHOME

# run this command to install all dependencies
RUN pip install -r requirements.txt

requirements.txt

pymilvus

and run docker build .

In this case I tried with python:3.12-alpine but I get the same result.

I am running Ubuntu 20.04 Docker version 26.1.4

[XuanYang-cn]

@onel THX for the scripts. The download of PyMilvus falls back to 2.4.1 because Milvus-lite donesn't support Apline Linux. And we intergrated Milvus-lite into pymilvus in 2.4.2. So that's why your codes raises errors about PyArrow. And PyArrow can't be compiled on Apline Linux too.

Please use python:3.12-slim for base. It works perfectly from my test.

[XuanYang-cn]

/unassign

[onel]

Thanks for the details. Should this be mentioned somewhere in the docs? I think alpine is the most used image for python development, it should be specified that -slim is recommended.

I can make a PR is you point to the right file