Open dsolerh opened 2 weeks ago
Try using --platform linux/amd64
for your docker build. I suspect that you're running on macOS and your build container has an aarch64 variant but the bitnami/pytorch
image doesn't, so you're getting a mismatch.
I tried that, and still get the same error
Does your build stage have an amd64 image available? Last I saw docker's platforming just fails silently when there's no matching platform, which is annoying. Can you share a minimal repro dockerfile?
I'm doing this and works (maybe there's a better way that'll remove the need to manually install rust, also the rust install part is just copied from the official image repo)
FROM python:3.11.6-slim
# Install Rust
ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH \
RUST_VERSION=1.79.0
RUN set -eux; \
apt-get update; \
apt-get install -y --no-install-recommends \
ca-certificates \
gcc \
libc6-dev \
wget \
; \
dpkgArch="$(dpkg --print-architecture)"; \
case "${dpkgArch##*-}" in \
amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='6aeece6993e902708983b209d04c0d1dbb14ebb405ddb87def578d41f920f56d' ;; \
armhf) rustArch='armv7-unknown-linux-gnueabihf'; rustupSha256='3c4114923305f1cd3b96ce3454e9e549ad4aa7c07c03aec73d1a785e98388bed' ;; \
arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='1cffbf51e63e634c746f741de50649bbbcbd9dbe1de363c9ecef64e278dba2b2' ;; \
i386) rustArch='i686-unknown-linux-gnu'; rustupSha256='0a6bed6e9f21192a51f83977716466895706059afb880500ff1d0e751ada5237' ;; \
*) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
esac; \
url="https://static.rust-lang.org/rustup/archive/1.27.1/${rustArch}/rustup-init"; \
wget "$url"; \
echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
chmod +x rustup-init; \
./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \
rm rustup-init; \
chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
rustup --version; \
cargo --version; \
rustc --version; \
apt-get remove -y --auto-remove \
wget \
; \
rm -rf /var/lib/apt/lists/*;
RUN pip install maturin
WORKDIR /tmp
COPY ./ ./
RUN maturin build --release
RUN pip install /tmp/target/wheels/*.whl
CMD ["python"] # I can run the image in a container and have my library available from the REPL
Now, the real thing:
FROM python:3.11.6-slim as rust-deps
# ... same logic as before except for the install of the package.
FROM bitnami/pytorch:2.1.1 as runtime
WORKDIR /app
COPY --from=rust-deps /tmp/target/wheels/*.whl /app/wheels/
RUN pip install --no-cache-dir /app/wheels/*.whl
CMD ["python"]
this second image I run it with docker build -t test-rust-pyo3 . && docker run --rm -it test-rust-pyo3
And you're saying that docker build --platform linux/amd64 -t test-rust-pyo3 . && docker run --rm -it test-rust-pyo3
fails with the error of aarch64 image? Can you paste output of the failure?
(Note I added --platform
flag to your command.)
#0 building with "desktop-linux" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.96kB done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.11.6-slim
#3 DONE 0.5s
#4 [internal] load metadata for docker.io/bitnami/pytorch:2.1.1
#4 DONE 0.5s
#5 [rust-deps 1/6] FROM docker.io/library/python:3.11.6-slim@sha256:cc758519481092eb5a4a5ab0c1b303e288880d59afc601958d19e95b300bc86b
#5 DONE 0.0s
#6 [runtime 1/4] FROM docker.io/bitnami/pytorch:2.1.1@sha256:2aa4a7367f4a888263985f677deb7dc22fb457fe1d9689872ac06e74f1ce442c
#6 DONE 0.0s
#7 [internal] load build context
#7 transferring context: 9.69kB done
#7 DONE 0.0s
#8 [rust-deps 3/6] RUN pip install maturin
#8 CACHED
#9 [rust-deps 2/6] RUN set -eux; apt-get update; apt-get install -y --no-install-recommends ca-certificates gcc libc6-dev wget ; dpkgArch="$(dpkg --print-architecture)"; case "${dpkgArch##*-}" in amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='6aeece6993e902708983b209d04c0d1dbb14ebb405ddb87def578d41f920f56d' ;; armhf) rustArch='armv7-unknown-linux-gnueabihf'; rustupSha256='3c4114923305f1cd3b96ce3454e9e549ad4aa7c07c03aec73d1a785e98388bed' ;; arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='1cffbf51e63e634c746f741de50649bbbcbd9dbe1de363c9ecef64e278dba2b2' ;; i386) rustArch='i686-unknown-linux-gnu'; rustupSha256='0a6bed6e9f21192a51f83977716466895706059afb880500ff1d0e751ada5237' ;; *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; esac; url="https://static.rust-lang.org/rustup/archive/1.27.1/${rustArch}/rustup-init"; wget "$url"; echo "${rustupSha256} *rustup-init" | sha256sum -c -; chmod +x rustup-init; ./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; rm rustup-init; chmod -R a+w $RUSTUP_HOME $CARGO_HOME; rustup --version; cargo --version; rustc --version; apt-get remove -y --auto-remove wget ; rm -rf /var/lib/apt/lists/*;
#9 CACHED
#10 [rust-deps 4/6] WORKDIR /tmp
#10 CACHED
#11 [rust-deps 5/6] COPY ./ ./
#11 DONE 0.0s
#12 [rust-deps 6/6] RUN maturin build --release
#12 0.944 Updating crates.io index
#12 1.501 Downloading crates ...
#12 1.635 Downloaded unindent v0.2.3
#12 1.687 Downloaded pyo3-macros v0.21.2
#12 1.716 Downloaded windows-targets v0.52.5
#12 1.737 Downloaded target-lexicon v0.12.14
#12 1.762 Downloaded unicode-ident v1.0.12
#12 1.855 Downloaded pyo3-macros-backend v0.21.2
#12 1.992 Downloaded portable-atomic v1.6.0
#12 2.021 Downloaded windows_x86_64_gnullvm v0.52.5
#12 2.072 Downloaded windows_i686_gnullvm v0.52.5
#12 2.122 Downloaded libc v0.2.155
#12 2.218 Downloaded windows_x86_64_msvc v0.52.5
#12 2.287 Downloaded windows_x86_64_gnu v0.52.5
#12 2.418 Downloaded windows_i686_msvc v0.52.5
#12 2.530 Downloaded windows_i686_gnu v0.52.5
#12 2.664 Downloaded windows_aarch64_msvc v0.52.5
#12 2.735 Downloaded regex-automata v0.4.7
#12 2.798 Downloaded pyo3 v0.21.2
#12 2.869 Downloaded windows_aarch64_gnullvm v0.52.5
#12 2.916 Downloaded syn v2.0.66
#12 2.960 Downloaded regex-syntax v0.8.4
#12 2.993 Downloaded regex v1.10.5
#12 3.041 Downloaded pyo3-ffi v0.21.2
#12 3.059 Downloaded memchr v2.7.4
#12 3.076 Downloaded rstest_macros v0.21.0
#12 3.086 Downloaded rstest v0.21.0
#12 3.104 Downloaded proc-macro2 v1.0.85
#12 3.113 Downloaded parking_lot_core v0.9.10
#12 3.120 Downloaded bitflags v2.5.0
#12 3.133 Downloaded aho-corasick v1.1.3
#12 3.154 Downloaded smallvec v1.13.2
#12 3.161 Downloaded semver v1.0.23
#12 3.170 Downloaded redox_syscall v0.5.1
#12 3.177 Downloaded quote v1.0.36
#12 3.185 Downloaded pyo3-build-config v0.21.2
#12 3.191 Downloaded parking_lot v0.12.3
#12 3.200 Downloaded once_cell v1.19.0
#12 3.209 Downloaded lock_api v0.4.12
#12 3.214 Downloaded scopeguard v1.2.0
#12 3.218 Downloaded rustc_version v0.4.0
#12 3.222 Downloaded relative-path v1.9.3
#12 3.226 Downloaded memoffset v0.9.1
#12 3.230 Downloaded autocfg v1.3.0
#12 3.236 Downloaded indoc v2.0.5
#12 3.243 Downloaded heck v0.4.1
#12 3.248 Downloaded glob v0.3.1
#12 3.252 Downloaded either v1.12.0
#12 3.257 Downloaded cfg-if v1.0.0
#12 3.358 🔗 Found pyo3 bindings
#12 3.944 🐍 Found CPython 3.11 at /usr/local/bin/python3
#12 3.945 📡 Using build options features from pyproject.toml
#12 4.988 Compiling target-lexicon v0.12.14
#12 4.992 Compiling once_cell v1.19.0
#12 4.996 Compiling autocfg v1.3.0
#12 4.999 Compiling proc-macro2 v1.0.85
#12 5.002 Compiling unicode-ident v1.0.12
#12 5.005 Compiling libc v0.2.155
#12 5.007 Compiling parking_lot_core v0.9.10
#12 5.010 Compiling smallvec v1.13.2
#12 5.018 Compiling cfg-if v1.0.0
#12 5.029 Compiling portable-atomic v1.6.0
#12 5.055 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.056 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.056 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.056 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.061 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.061 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.064 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.064 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.067 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.069 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.071 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.071 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.075 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.076 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.100 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.100 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.142 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.143 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 5.150 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 5.150 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 6.220 Compiling scopeguard v1.2.0
#12 6.322 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 6.325 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 6.863 Compiling heck v0.4.1
#12 6.918 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 6.918 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 8.135 Compiling either v1.12.0
#12 8.219 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 8.232 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 8.266 Compiling indoc v2.0.5
#12 8.354 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 8.355 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 9.138 Compiling unindent v0.2.3
#12 9.249 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 9.249 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 10.33 Compiling lock_api v0.4.12
#12 10.34 Compiling memoffset v0.9.1
#12 10.42 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 10.43 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 10.46 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 10.46 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 10.55 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 10.55 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 10.59 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 10.59 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 11.13 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 11.14 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 11.28 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 11.28 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 13.13 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 13.13 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 13.13 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 13.13 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 14.32 Compiling quote v1.0.36
#12 14.36 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 14.36 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 14.36 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 14.36 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 14.38 Compiling pyo3-build-config v0.21.2
#12 14.43 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 14.43 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 16.28 Compiling syn v2.0.66
#12 16.33 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 16.33 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 16.50 Compiling parking_lot v0.12.3
#12 16.54 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 16.54 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 19.82 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 19.82 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 25.16 Compiling pyo3-ffi v0.21.2
#12 25.16 Compiling pyo3 v0.21.2
#12 25.19 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 25.19 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 25.19 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 25.19 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 27.57 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 27.57 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 30.89 Compiling pyo3-macros-backend v0.21.2
#12 30.92 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 30.92 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 43.51 Compiling pyo3-macros v0.21.2
#12 43.55 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 43.55 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 46.88 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 46.88 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 75.59 Compiling my_rust_code v0.1.0 (/tmp)
#12 75.63 <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
#12 75.63 <jemalloc>: (This is the expected behaviour if you are running under QEMU)
#12 85.87 Finished `release` profile [optimized] target(s) in 1m 21s
#12 88.91 📦 Built wheel for CPython 3.11 to /tmp/target/wheels/my_rust_code-0.1.0-cp311-cp311-manylinux_2_34_x86_64.whl
#12 DONE 89.0s
#13 [runtime 2/4] WORKDIR /app
#13 CACHED
#14 [runtime 3/4] COPY --from=rust-deps /tmp/target/wheels/*.whl /app/wheels/
#14 DONE 0.0s
#15 [runtime 4/4] RUN pip install --no-cache-dir /app/wheels/*.whl
#15 2.724 DEPRECATION: Loading egg at /opt/bitnami/python/lib/python3.11/site-packages/pip-23.3.1-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
#15 2.769 ERROR: my_rust_code-0.1.0-cp311-cp311-manylinux_2_34_x86_64.whl is not a supported wheel on this platform.
#15 4.618
#15 4.618 [notice] A new release of pip is available: 23.3.1 -> 24.1.2
#15 4.618 [notice] To update, run: pip install --upgrade pip
#15 ERROR: process "/bin/bash -o errexit -o nounset -o pipefail -c pip install --no-cache-dir /app/wheels/*.whl" did not complete successfully: exit code: 1
------
> [runtime 4/4] RUN pip install --no-cache-dir /app/wheels/*.whl:
2.724 DEPRECATION: Loading egg at /opt/bitnami/python/lib/python3.11/site-packages/pip-23.3.1-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation.. Discussion can be found at https://github.com/pypa/pip/issues/12330
2.769 ERROR: my_rust_code-0.1.0-cp311-cp311-manylinux_2_34_x86_64.whl is not a supported wheel on this platform.
4.618
4.618 [notice] A new release of pip is available: 23.3.1 -> 24.1.2
4.618 [notice] To update, run: pip install --upgrade pip
------
Dockerfile:54
--------------------
52 | COPY --from=rust-deps /tmp/target/wheels/*.whl /app/wheels/
53 |
54 | >>> RUN pip install --no-cache-dir /app/wheels/*.whl
55 |
56 | CMD ["python"]
--------------------
ERROR: failed to solve: process "/bin/bash -o errexit -o nounset -o pipefail -c pip install --no-cache-dir /app/wheels/*.whl" did not complete successfully: exit code: 1
Ok great, so it seems pretty clear that the docker --platform
is not the issue.
I think the next thing to try would be the manylinux version. The 2_34
part of the wheel platform is the glibc version coming from your build image. My next guess would be that the bitnami
stage has an older glibc version.
yes, the versions differ from each other, thanks for the tip, if there a way to specify on maturin
/setuptools-rust
that the package can be compatible with different glibc?
I think the problem is that maturin can't guarantee the build artefacts haven't linked against functionality in the newer glibc.
The simplest solution might be to use one of the manylinux images as the base of your build image instead of python-slim
.
Hi, I have a package in rust using PyO3 that I need to build inside a docker container as part of a multi-stage build and pass it to the
runtime
stage for install and execution (I have this cumbersome setup as a requirement). I tried to usematurin
by runningmaturin build --release
but in theruntime
stage when I try to usepip install my-package-0.1.0-cp311-cp311-manylinux_2_34_aarch64.whl
I get this error:ERROR: my-package-0.1.0-cp311-cp311-manylinux_2_34_aarch64.whl is not a supported wheel on this platform
. The python version in which I'm building the wheel is 3.11.6 and it's the same version used in theruntime
. The theruntime
is running from bitnami/pytorch:2.1.1, I whould appretiate any advice to setup the Dockerfile to make this. PS: Sorry if I'm not more precise in the description, but I'm bounded by an NDA agreement.