ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

[TheOneTrueGuy]

Trying a fresh copy of the colab notebook and getting the above error.

the complete trace:

Building wheels for collected packages: fairscale, tokenizers, sacremoses Building wheel for fairscale (pyproject.toml) ... done Created wheel for fairscale: filename=fairscale-0.4.4-py3-none-any.whl size=292831 sha256=590e9e184d72ea0a0bbafe8f98dc19e3cd9dbd421660c1fcd2e76a73a25afa46 Stored in directory: /root/.cache/pip/wheels/08/58/6f/56c57fa8315eb0bcf0287b580c850845be5f116359b809e9f1 error: subprocess-exited-with-error

× Building wheel for tokenizers (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. Building wheel for tokenizers (pyproject.toml) ... error ERROR: Failed building wheel for tokenizers Building wheel for sacremoses (setup.py) ... done Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895240 sha256=01ac150702a4da6cba5a2ea244e3e480662a1335e06f072e6996cafcafc486bc Stored in directory: /root/.cache/pip/wheels/00/24/97/a2ea5324f36bc626e1ea0267f33db6aa80d157ee977e9e42fb Successfully built fairscale sacremoses Failed to build tokenizers ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

[hoangpnhat]

Maybe this problem about Rust config. In my case, in linux systerm: Install Rust curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh after finish source "$HOME/.cargo/env" and pip install -r requirement.txt

[h1de0us]

In my case the problem was with Deny-by-default lints in Rust. The following commands helped: Allow warnings: export RUSTFLAGS='-A warnings' Allow invalid reference casting: export RUSTFLAGS='-A invalid_reference_casting' and pip install -r requirements.txt

[nikola-rahman]

Steps that worked for me in Colab:

1. Install Rust toolchain:

   !curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

2. Export path to Rust for every !pip install. In notebooks !command spawns a new terminal every time which doesn't have rust path in $PATH variable. So I just do it for every !pip install. In the notebook where you install stuff, change this line to:

   !export PATH="$HOME/.cargo/bin:$PATH" && pip3 install -v transformers==4.15.0 timm==0.4.12 fairscale==0.4.4

Maybe there is a more elegant way to prepend rust path to $PATH for every new terminal (.bashrc or whatever colab is using) but I didn't play around with it.

[JiahuiLiu23356]

我降低了python版本到3.8，解决了这个问题https://blog.csdn.net/B1gBigDream/article/details/132718300