huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.68k stars 746 forks source link

Python3.12 build for Windows is not available #1429

Closed ghostplant closed 5 months ago

ghostplant commented 5 months ago

On Windows platforms using Python3.12 x64. The following installation fails:

$ pip install tokenizers
Collecting tokenizers
  Downloading tokenizers-0.15.0.tar.gz (318 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 318.5/318.5 kB 858.8 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]

      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/

      Checking for Rust toolchain....
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
ArthurZucker commented 5 months ago

Sorry a release is long due for python 3.12, will get to it tomorrow hopefully

ghostplant commented 5 months ago

Any progress? Still no window build for Python 3.12. (https://pypi.org/project/tokenizers/#files) Standard Windows doesn't have rust installed so building from source is not a common option.

ArthurZucker commented 5 months ago

We released yesterday 🥳 🤗