Error while installing in Docker dev environment

[ershovio]

I tried to install tiktoken in Docker dev environment with Python 3.9 using the default approach: pip install tiktoken But I got an error: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects Here is the stack trace:

root ➜ /com.docker.devenvironments.code (master ✗) $ pip install tiktoken
Collecting tiktoken
  Using cached tiktoken-0.3.0.tar.gz (24 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting blobfile>=2
  Using cached blobfile-2.0.1-py3-none-any.whl (73 kB)
Collecting requests>=2.26.0
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting regex>=2022.1.18
  Using cached regex-2022.10.31-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (769 kB)
Requirement already satisfied: lxml~=4.9 in /usr/local/lib/python3.9/dist-packages (from blobfile>=2->tiktoken) (4.9.2)
Requirement already satisfied: urllib3<3,>=1.25.3 in /usr/local/lib/python3.9/dist-packages (from blobfile>=2->tiktoken) (1.26.13)
Requirement already satisfied: filelock~=3.0 in /usr/local/lib/python3.9/dist-packages (from blobfile>=2->tiktoken) (3.8.2)
Collecting pycryptodomex~=3.8
  Using cached pycryptodomex-3.17-cp35-abi3-manylinux2014_aarch64.whl (2.1 MB)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests>=2.26.0->tiktoken) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests>=2.26.0->tiktoken) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests>=2.26.0->tiktoken) (2022.9.24)
Building wheels for collected packages: tiktoken
  Building wheel for tiktoken (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tiktoken (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [37 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-39
      creating build/lib.linux-aarch64-cpython-39/tiktoken
      copying tiktoken/registry.py -> build/lib.linux-aarch64-cpython-39/tiktoken
      copying tiktoken/__init__.py -> build/lib.linux-aarch64-cpython-39/tiktoken
      copying tiktoken/model.py -> build/lib.linux-aarch64-cpython-39/tiktoken
      copying tiktoken/core.py -> build/lib.linux-aarch64-cpython-39/tiktoken
      copying tiktoken/load.py -> build/lib.linux-aarch64-cpython-39/tiktoken
      creating build/lib.linux-aarch64-cpython-39/tiktoken_ext
      copying tiktoken_ext/openai_public.py -> build/lib.linux-aarch64-cpython-39/tiktoken_ext
      running egg_info
      writing tiktoken.egg-info/PKG-INFO
      writing dependency_links to tiktoken.egg-info/dependency_links.txt
      writing requirements to tiktoken.egg-info/requires.txt
      writing top-level names to tiktoken.egg-info/top_level.txt
      reading manifest file 'tiktoken.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'Makefile'
      adding license file 'LICENSE'
      writing manifest file 'tiktoken.egg-info/SOURCES.txt'
      copying tiktoken/py.typed -> build/lib.linux-aarch64-cpython-39/tiktoken
      running build_ext
      running build_rust
      error: can't find Rust compiler

      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

      To update pip, run:

          pip install --upgrade pip

      and then retry package installation.

      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tiktoken
Failed to build tiktoken
ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects

[hauntsaninja]

Thanks, duplicate of https://github.com/openai/tiktoken/issues/52

[ershovio]

@hauntsaninja you referenced my issue (duplicate of #52) but 52 it's this issue

[hauntsaninja]

Oops, duplicate of https://github.com/openai/tiktoken/issues/23

The relevant bit of the error message is error: can't find Rust compiler. The solution is either install a rust compiler in your image or install from a wheel. I do not currently ship ARM64 wheels, that's what the duplicate issue is about (but you could build your own)