Publish Python 3.12 wheels on PyPI

[mayeut]

Is your feature request related to a problem? Please describe. For now, installing python-crc32 with Python 3.12 is done by building from sources rather than just installing a wheel.

Describe the solution you'd like Please publish Python 3.12 wheels.

Describe alternatives you've considered Building from sources: this is slow & error prone.

Additional context Python 3.12 will be released next week and has been ABI stable since rc1.

[parthea]

Blocked by #149

[bricker]

@parthea how can I help get this done?

[martin-traverse]

We have a multi-cloud product and need this to support GCP with Python 3.12. The current situation is that Python 3.12 is supported on AWS, but not on GCP or Azure because of dependency issues. If there is anything I can do to help, please let me know.

[dsotirho-ucsc]

This is a blocking us from upgrading to Python 3.12. Any news on this issue would be most appreciated. Thanks!

[ADR-007]

As a workaround you can build it manually in Docker using this instruction (+several hours for debugging 😀):

ENV PY_BIN=$VIRTUAL_ENV/bin/python
RUN git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c \
    && cd python-crc32c  \
    && pip install -r scripts/dev-requirements.txt \
    && ./scripts/local-linux/build_libcrc32c.sh \
    && pip install --no-index --find-links=wheels google-crc32c \
    && python ./scripts/check_crc32c_extension.py \
    && python -c "from google_crc32c import *" \
    && ls --hide=usr | xargs -d '\n' rm -rf

[edgarrmondragon]

As a workaround you can build it manually in Docker using this instruction (+several hours for debugging 😀):

ENV PY_BIN=$VIRTUAL_ENV/bin/python
RUN git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c \
    && cd python-crc32c  \
    && pip install -r scripts/dev-requirements.txt \
    && ./scripts/local-linux/build_libcrc32c.sh \
    && pip install --no-index --find-links=wheels google-crc32c \
    && python ./scripts/check_crc32c_extension.py \
    && python -c "from google_crc32c import *" \
    && ls --hide=usr | xargs -d '\n' rm -rf

Not really a workaround when the request is specifically to avoid having to build from source 😉

[martin-traverse]

As a workaround you can build it manually in Docker using this instruction (+several hours for debugging 😀):

ENV PY_BIN=$VIRTUAL_ENV/bin/python

RUN git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c \

    && cd python-crc32c  \

    && pip install -r scripts/dev-requirements.txt \

    && ./scripts/local-linux/build_libcrc32c.sh \

    && pip install --no-index --find-links=wheels google-crc32c \

    && python ./scripts/check_crc32c_extension.py \

    && python -c "from google_crc32c import *" \

    && ls --hide=usr | xargs -d '\n' rm -rf

I do appreciate you trying to offer a solution, but honestly this is a really bad developer experience. If it was just me I wouldn't mind, but our users are modellers, they install our library using pip and expect it to work. We can't start publishing instructions telling them to install Docker and build stuff from source, this is exactly the kind of pain we are trying to solve for them. Also the tools available in enterprise environments are normally hugely restricted, if Docker is even an option you probably have to raise a bunch of requests and go through an approval workflow which can take an unlimited amount of time, maybe you even need an entirely different machine and have to set up all your tools again. It's just not practical and our users won't do it.

We will have to continue telling our users that GCP doesn't support Python 3.12 until the packages are available through the normal mechanism. This is what we need:

pip install google-auth
pip install google-cloud-core
pip install google-cloud-storage

I hope this helps explain the context a little bit! In my field there are lots of people who want to use these tools but issues like this will stop them in their tracks.

[edgarrmondragon]

Gentle bump. Python 3.12 is now 6 months old.

[parthea]

See https://github.com/googleapis/python-crc32c/issues/149#issuecomment-2000785692 for ETA

[jlaine]

Could you conceivably make use of Python's limited API [1]? This would avoid having to rebuild wheels whenever a new version of Python is released.

[1] https://docs.python.org/3/c-api/stable.html#limited-c-api

[hannes-ucsc]

Why is #149 a blocker?

This appears to be affecting everyone attempting to use the Google Cloud SDK with Python 3.12, and should therefore be expedited, no?

[hannes-ucsc]

@ADR-007

As a workaround you can build it manually in Docker …

Thank you!

[adamchainz]

To put some practical numbers on the message “significantly slower”, I ran a mini benchmark, checksumming 1MiB of data on both versions.

Python 3.11, with extension:

In [1]: import google_crc32c

In [2]: %timeit cs = google_crc32c.Checksum() ; cs.update(b'1' * 1024 * 1024) ; cs.digest()
244 µs ± 4.62 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Python 3.12, without extension:

In [1]: import google_crc32c
/.../google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported, `google-crc32c` is using a pure python implementation that is significantly slower. If possible, please configure a c build environment and compile the extension
  warnings.warn(_SLOW_CRC32C_WARNING, RuntimeWarning)

In [2]: %timeit cs = google_crc32c.Checksum() ; cs.update(b'1' * 1024 * 1024) ; cs.digest()
125 ms ± 1.63 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

It’s about ~500,000 times slower. Perhaps one could budget an extra 100ms for each Google API request.

This degradation is not tolerable for the project I’m working on, so we’ll have to delay upgrading Python or workaround and compile ourselves.

[hannes-ucsc]

This bash command builds the wheels and drops them in the current directory:

for arch in arm64 amd64
do docker run --rm -iv "$PWD":/host --platform=linux/$arch python:3.12 /bin/bash <<-"END"
    set -ex
    export PY_BIN="$(which python)"
    git clone --recursive --depth 1 --branch v1.5.0 https://github.com/googleapis/python-crc32c
    cd python-crc32c
    pip install -r scripts/dev-requirements.txt
    ./scripts/local-linux/build_libcrc32c.sh
    pip install --no-index --find-links=wheels google-crc32c
    python ./scripts/check_crc32c_extension.py
    python -c "from google_crc32c import *"
    cp wheels/*.whl /host
END
done

Based on https://github.com/googleapis/python-crc32c/issues/178#issuecomment-2001053600 by @ADR-007

[andrewsg]

This work is in progress, we're having some build system issues on Windows and Apple Silicon. Hope to have an update soon. Thanks all.