search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.52k stars 856 forks source link

Build wheels for Python 3.13 #346

Closed iisakkirotko closed 1 month ago

iisakkirotko commented 1 month ago

Hi! With python 3.13 coming out (hopefully) later today, it would be nice to have wheels for the new version available.

This PR adds Python 3.13 to the versions for which tiktoken should be built (for now together with CIBW_PRERELEASE_PYTHONS: True, since only 3.13.0rc2 is available to date). It also bumps pyo3 to version 0.22 which supports Python 3.13, together with the gil-refs feature flag for backwards compatibility.

I suppose a discussion could be had about whether support for Python 3.8 should be dropped due to its end-of-life, but that's for another day.

Let me know if there's anything else that should be accounted for, my experience with Rust is very limited :)

hauntsaninja commented 1 month ago

Thanks for the PR! The internal version already uses the new PyO3 Bound API, I'll sync the codebase. Not sure what's regressed the QEMU aarch64 builds

hauntsaninja commented 1 month ago

tiktoken 0.8 ships 3.13 wheels. Needed a few things to be different from this diff. Thanks again!