ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects

[2anirban]

Was trying to upload the project in Google Cloud run..using python 3.11.5 Not able to build wheels for Tiktoken..Tried every version. Can anybody help.

"Building wheel for tiktoken (pyproject.toml): started Building wheel for tiktoken (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error

? Building wheel for tiktoken (pyproject.toml) did not run successfully. ? exit code: 1 ??> [37 lines of output] running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-cpython-312 creating build/lib.linux-x86_64-cpython-312/tiktoken copying tiktoken/load.py -> build/lib.linux-x86_64-cpython-312/tiktoken copying tiktoken/core.py -> build/lib.linux-x86_64-cpython-312/tiktoken copying tiktoken/registry.py -> build/lib.linux-x86_64-cpython-312/tiktoken copying tiktoken/init.py -> build/lib.linux-x86_64-cpython-312/tiktoken copying tiktoken/model.py -> build/lib.linux-x86_64-cpython-312/tiktoken creating build/lib.linux-x86_64-cpython-312/tiktoken_ext copying tiktoken_ext/openai_public.py -> build/lib.linux-x86_64-cpython-312/tiktoken_ext running egg_info writing tiktoken.egg-info/PKG-INFO writing dependency_links to tiktoken.egg-info/dependency_links.txt writing requirements to tiktoken.egg-info/requires.txt writing top-level names to tiktoken.egg-info/top_level.txt reading manifest file 'tiktoken.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'Makefile' adding license file 'LICENSE' writing manifest file 'tiktoken.egg-info/SOURCES.txt' copying tiktoken/py.typed -> build/lib.linux-x86_64-cpython-312/tiktoken running build_ext running build_rust error: can't find Rust compiler

  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

  To update pip, run:

      pip install --upgrade pip

  and then retry package installation.

  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for tiktoken Building wheel for yarl (pyproject.toml): started Building wheel for yarl (pyproject.toml): finished with status 'done' Created wheel for yarl: filename=yarl-1.9.2-cp312-cp312-linux_x86_64.whl size=285235 sha256=74068f72957a1a126a9b7b237663c56f66563b61765f0308972e87cf6c4d4be2 Stored in directory: /root/.cache/pip/wheels/84/e3/6a/7d0fa1abee8e4aa39922b5bd54689b4b5e4269b2821f482a32 Building wheel for multidict (pyproject.toml): started Building wheel for multidict (pyproject.toml): finished with status 'done' Created wheel for multidict: filename=multidict-6.0.4-cp312-cp312-linux_x86_64.whl size=114928 sha256=8b8c43433d60abce6a0995ad8d0be737bb757d67a6181f9137a0054d8d273e91 Stored in directory: /root/.cache/pip/wheels/f6/d8/ff/3c14a64b8f2ab1aa94ba2888f5a988be6ab446ec5c8d1a82da Successfully built bs4 frozenlist yarl multidict Failed to build tiktoken ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects

[notice] A new release of pip is available: 23.2.1 -> 23.3.1 [notice] To update, run: pip install --upgrade pip The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1 ERROR ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1 "

[mohaned-dewedar]

Running into this same issue right now.

[ValereDemelierNepa]

Running into the same issue as well

[jonomacx]

I had code that was working yesterday when I was on Python 3.10, today I have upgraded to 3.12 and am using a virtualenv and I am now hitting the same issue. My issue was the rust compiler was missing. I installed via https://rustup.rs/ and it's working now. Odd that I had never installed it yesterday in the first place?

[TJaenichen]

Can confirm, installed from the rust website, restarted the command line and it works (win 10)

[TJaenichen]

Like I wrote, installing rust fixed it for me.

[hauntsaninja]

OP's problem is that they're not using Python 3.11, they're using Python 3.12. tiktoken doesn't yet have 3.12 wheels uploaded to PyPI, so you'll fall back to sdist and compilation, which is why you would need Rust compiler. I'll add 3.12 wheels soon.

Closing as a duplicate of https://github.com/openai/tiktoken/issues/205

[Yunhua468]

OP's problem is that they're not using Python 3.11, they're using Python 3.12. tiktoken doesn't yet have 3.12 wheels uploaded to PyPI, so you'll fall back to sdist and compilation, which is why you would need Rust compiler. I'll add 3.12 wheels soon.

Closing as a duplicate of #205

Down grade python version from 3.12 to 3.11, it works for me.

[hauntsaninja]

tiktoken 0.5.2 and newer provide Python 3.12 wheels on PyPI