Failed building wheel for tokenizers

[ks-kasra]

Hi, I got an error after pip install. I don't know where the problem comes from!

I installed pyproject.toml and rust but I still have this error

ERROR :

                  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
                  [end of output]
          note: This error originates from a subprocess, and is likely not a problem with pip.
          ERROR: Failed building wheel for tokenizers
  Successfully built tortoise-tts
  Failed to build tokenizers
  ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

[valkanesko]

I am getting a similar error:

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
      [end of output]
        note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Successfully built tortoise-tts
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

It originates from here:

      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
          |
      522 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      526 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default

      warning: `tokenizers` (lib) generated 3 warnings
      error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted

[Morac2]

I got the same problem on an Apple Silicon Mac with rustc 1.76.0 installed with HomeBrew.

I found this page which linked to the follow issue in tokenizers, but that was closed as working and it mentioned rust 1.75: https://github.com/huggingface/tokenizers/issues/1359

A work around is to set the RUSTFLAGS environment variable to "-A invalid_reference_casting" before running pip install.

export RUSTFLAGS="-A invalid_reference_casting"

After I did that, I could install successfully.

Looking at that page though it appears the actual fix is to upgrade to a newer version of tokenizer.

[sleeplessKomodo]

Looking at that page though it appears the actual fix is to upgrade to a newer version of tokenizer.

yep

conda update tokenizers

then

conda install transformers==4.31.0

allowed me to install all dependencies for tortoise-tts==3.0.0

[BirgitPohl]

I got it too. Mac M2.

[sai-147]

i also got same issue

[DerekTDLe]

Same issue

[robbiemu]

so the issue now is that tokenizers is at 0.20 and this is depending on tokenizers 0.13, which was never supporting more modern versions of rust nor python 3.12

I think we can update the setup.py to 'tokenizers>=0.20.0', 'transformers==4.45.1', currently ?

edit:

seems so, build finished, with just also 'numpy>=1.22,<2.1,!=1.22.2,!=1.22.1,!=1.22.0', to satisfy both librosa and numba

edit2: I am generating audio :)

PS, Id be happy to provide a PR but I see a lot of open ones, Im thinking you might have bigger fish to fry right now and this is really a very simple change -- and of course I only have tested in a current Mac environment.

[MarRazane]

i have the same issue, and i couldn't install transformers4rec