manylinux wheel doesn't work on Centos 7 / Python 3.8.2

[AvdN]

[root@vh0 ~]# /opt/python/3.8.2/bin/python -m venv /opt/util/py38
[root@vh0 ~]# source /opt/util/py38/bin/activate
(py38) [root@vh0 ~]# pip install blake3
Collecting blake3
  Using cached https://files.pythonhosted.org/packages/d7/d9/4740acd52573498efa4beddb98b3e4ecfe6f4a8c425e8c618fb2092cd802/blake3-0.1.4-cp38-cp38-manylinux1_x86_64.whl
Installing collected packages: blake3
Successfully installed blake3-0.1.4
(py38) [root@vh0 ~]# python
Python 3.8.2 (default, Feb 26 2020, 08:29:54) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import blake3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /opt/util/py38/lib/python3.8/site-packages/blake3.cpython-38-x86_64-linux-gnu.so)
>>> exit()
(py38) [root@vh0 ~]# cat /etc/system-release
CentOS Linux release 7.7.1908 (Core)

I searched for the error message, which might be similar to https://github.com/PyO3/maturin/issues/47 referenced in https://github.com/ijl/orjson/issues/2 . I could not determine if that is indeed adressed in rust or Pyo3

[AvdN]

I was able to solve this by compiling locally. In case this is not fixed in the .whl yet, the steps for this are

• deinstall rust and/or cargo if installed with yum (yum remove rust cargo)
• install rustup: curl https://sh.rustup.rs -sSf | sh
• make sure python3 is in your path (e.g. virtualenv)
• clone repo: git clone https://github.com/oconnor663/blake3-py.git
• cd blake3-py
• rustup show (not sure if there is a better subcommand, this seemed to work, rustup is not very intuitiv, nor well documented for first time rust users)
• cargo build --release
• cp target/release/libblake3.so /path/to/your/python/lib/python*/site-packages/blake3.so

[oconnor663]

Thanks for the thorough report. This is my first project with binary wheels, so I'm learning as I go, and I wouldn't be at all surprised if I've missed something important. Do you think this could be solved by building the Linux wheels inside a CentOS Docker container? Or, maybe this issue would go away if we built against musl instead of glibc?

[AvdN]

No, I don't think that will help, for two reasons:

• AFAIK there is no way for pip install to select a name like onblake3-0.1.4-cp38-cp38-centos7_x86_64.whl on the basis that you would be running CentOS
• The whole idea of manylinux1 is to provide a compile environment that has older libraries (including glibc) that are compatible with all supported/current LInuxes (as the newer libraries are backwards compatible).

I suspect that the Dockerfile used for blake3-py is updated with rust and some other stuff that negates the second point, by installing (and using glibc 2.18). The aging CentOS7 still uses 2.17. If that is so, then looking at manylinux2010 would be of no help. In my project using manylinux I use the pre-installed compiler and libraries and that doesn't give a problem on Centos ( https://sourceforge.net/p/ruamel-yaml-clib/code/ci/default/tree/ ).

Rust can use an older version of glibc (because otherwise I would not have been able to compile using rustup/cargo), but maybe not as old as the pre-installed libraries in manylinux1 containers.

Where is your Dockerfile, it is not part of the project, or is that magically created on the fly by one of the configuration files? Can you share it so I can have a look at it?

[oconnor663]

I don't maintain a Dockerfile in this project. Currently just run maturin build and take what it gives me. That's done via:

• https://github.com/oconnor663/blake3-py/blob/master/.github/workflows/tag.yml
• https://github.com/oconnor663/blake3-py/blob/master/.github/workflows/maturin_build_wheel.py

That is, whenever I push a tag, it builds a wheel on each CI target platform and uploads it as a tag asset on GitHub. Finally, if all those builds succeed, another job fetches each tag asset and uploads them to PyPI. Here's an example of one of these packaging CI runs: https://github.com/oconnor663/blake3-py/actions/runs/66240625

[AvdN]

maturin has no home-page link on their PyPI page, but fortunately the page includes a reference to the docker container konstin2/maturin and the page for that has a link to a github project with a Dockerfile.

I built a container from that file and expected its contents. It only has libc-2.5.so so that looks good, but then I found this link (that konstin, is the provider of maturin) from which I gathered some decisions are made at link time, and although things work for blake3-py when compiled on Centos7 with libc 2.17, they don't work if compiled on libc 2.5 (from CentOS 5 in the manylinux container) and then are run on CentOS 7 with libc 2.17

It would be good for maturin to document the restrictions in the documentation, to prevent false expectations.

It might be easier to recommend building/installing libc 2.18 on CentOS 7... Anyway this is typical case of "Everything you never wanted to know about rust, but were forced to find out", when I just wanted to use blake3 from python because it is so much faster than the sha224 I have been using.

[oconnor663]

I've linked to this issue from the README.

[oconnor663]

Here's how I would install blake3-py in a CentOS7 Docker container, as an example. Readers will want to tweak this to match their particular use case (for example, if Python is installed under some other path). First, to start the container, I run:

$ docker run -ti centos:centos7

Then inside the container (where I'm root, so I don't need sudo for anything), I run:

# You likely have some of these installed already, but the Docker container doesn't.
$ yum install -y git python3 gcc

# Install nightly Rust. Note that if you already have rustup installed, the Cargo
# command below will automatically download the nightly toolchain (because of the
# rust-toolchain file in the repo), and you can skip this part.
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > /tmp/rustup.sh
$ sh /tmp/rustup.sh --default-toolchain nightly --profile minimal -y

# Get a copy of blake3-py. You could also download a release tarball.
$ git clone https://github.com/oconnor663/blake3-py /tmp/blake3-py
$ cd /tmp/blake3-py

# Build the shared library. I'm using an absolute path to Cargo, because although
# rustup edited my ~/.profile to add ~/.cargo/bin to my $PATH above, I haven't yet
# restarted the shell or sourced ~/.profile to make that take effect.
$ ~/.cargo/bin/cargo build --release

# Install the shared library. You will need to tweak this for the Python version
# and install path you're using.
$ cp target/release/libblake3.so /usr/lib/python3.6/site-packages/blake3.so

$ python3
Python 3.6.8 (default, Apr  2 2020, 13:34:55) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import blake3
>>> blake3.__file__
'/usr/lib/python3.6/site-packages/blake3.so'
>>> blake3.blake3(b"foo").hexdigest()
'04e0bb39f30b1a3feb89f536c93be15055482df748674b00d26e5a75777702e9'

[oconnor663]

It might be easier to recommend building/installing libc 2.18 on CentOS 7

Is there a supported way to go about that, without the risk of breaking all sorts of programs on the machine?

[AvdN]

It might be easier to recommend building/installing libc 2.18 on CentOS 7

Is there a supported way to go about that, without the risk of breaking all sorts of programs on the machine?

I don't think that is a problem, as long you leave the link /lib/libc.so.6 pointing to libc-2.17.so, although I am not sure if that is enough to convince rust to use the 2.18 version.

I am using manylinux1 for my ruamel.yaml.clib wheels without problem, but maybe that is because I only need to compile C code that is as about as old CentOS 5.

If you start with a manylinux1 container, you can successfully build blake3 and run it, with some minor changes (older curl, python3 not in PATH) to your steps for the CentOS 7 container.

If you create a Dockerfile with the following content:

FROM quay.io/pypa/manylinux1_x86_64:latest

MAINTAINER Anthon van der Neut <a.van.der.neut@ruamel.eu>

# based on https://github.com/oconnor663/blake3-py/issues/5#issuecomment-621412056

# Install nightly Rust. Note that if you already have rustup installed, the Cargo
# command below will automatically download the nightly toolchain (because of the
# rust-toolchain file in the repo), and you can skip this part.
RUN curl -sSf https://sh.rustup.rs > /tmp/rustup.sh
RUN sh /tmp/rustup.sh --default-toolchain nightly --profile minimal -y

# Get a copy of blake3-py. You could also download a release tarball.
RUN git clone https://github.com/oconnor663/blake3-py /src

RUN mkdir /dist

RUN echo 'cd /src' > /usr/bin/makewheel
RUN echo 'rm -f /tmp/*.whl'                               >> /usr/bin/makewheel
RUN echo 'for PYVER in $*; do'                            >> /usr/bin/makewheel
RUN echo '  for PYBIN in /opt/python/cp$PYVER*/bin/; do'  >> /usr/bin/makewheel
RUN echo '     echo "$PYBIN"'                             >> /usr/bin/makewheel
RUN echo '     ${PYBIN}/pip install -Uq pip'            >> /usr/bin/makewheel
# Build the shared library. I'm using an absolute path to Cargo, because although
# rustup edited my ~/.profile to add ~/.cargo/bin to my $PATH above, I haven't yet
# restarted the shell or sourced ~/.profile to make that take effect.
# the PYBIN PATH is needed for pyo3 to function
RUN echo '     PATH=${PYBIN}:${PATH} /root/.cargo/bin/cargo build --release' >> /usr/bin/makewheel
RUN echo '     PATH=/root/.cargo/bin:${PYBIN}:${PATH}  ${PYBIN}/pip wheel . -w /dist' >> /usr/bin/makewheel
RUN echo '  done'                                         >> /usr/bin/makewheel
RUN echo 'done'                                           >> /usr/bin/makewheel
RUN echo ''                                               >> /usr/bin/makewheel
RUN chmod 755 /usr/bin/makewheel

CMD /usr/bin/makewheel 35 36 37 38

And then run:

mkdir dist
docker build -t blake3
docker run --mount=type=bind,source=$(pwd)/dist,target=/dist blake3:latest

You'll end up with .whl files in dist for 3.5-3.8 that work on CentOS 7 (tested python 3.7 and 3.8) and should work on CentOS 5 & 6 (if someone bothered to compile Python3 from source for that). The wheels are called manylinux1 (where I would expect manylinux2010). That looks like much simpler solution than the requirement of installing libc-2.18

[oconnor663]

So wait, have I just been doing everything wrong? :sweat_smile: I thought maturin build took care of these details? But now I see this:

For full manylinux compliance you need to compile in a cent os 5 docker container.

I guess I'll try to integrate the Docker commands they have in the docs there with what I'm doing in CI. It sounds like that might just fix all the issues we're having? It's unclear to me how much of this was clear to you before, and how much you're learning at the same time as me :)

[AvdN]

Yes I thought that was what maturin was doing, but since I use manylinux1 myself directly that is what I tried. And that is what works with both manylinux1 and manylinux2010. I'll update the comment above.

I would just include the docker command and the Dockerfile in your build process directly.

[oconnor663]

I've just released version 0.1.5, which is built properly in the manylinux Docker container recommended by the Maturin docs. Testing it by hand in a CentOS 7 container seems to work, so hopefully this issue is resolved. Please let me know if you get a chance to test it yourself. Many thanks for reporting it and for helping me get it fixed.

[AvdN]

I installed and tested in 3.5, 3.6, 3.7, 3.8, and they all worked.

The only hickup was that for Python 3.5 I got a message

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-nah1dj7b/blake3

which was caused by its old pip version not knowing about wheel files. I had to do pip install -Uq pip first and then the wheel instead of the .tar.gz was taken.

Thanks for fixing this.