[Bug]: issue with tokenizers

projecthorizon993 commented 3 months ago

Checklist

[x] The issue exists after disabling all extensions
[x] The issue exists on a clean installation of webui
[ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
[x] The issue exists in the current version of the webui
[x] The issue has not been reported before recently
[x] The issue has been reported before but has not been fixed yet

What happened?

can't open because of the tokenizers can't build

Steps to reproduce the problem

idk it just pop up

What should have happened?

it run as usual but now it bonked

What browsers do you use to access the UI ?

No response

Sysinfo

from my console can't open the webUI

Console logs

Traceback (most recent call last):
  File "/workspace/file/stable-diffusion-webui/launch.py", line 48, in <module>
    main()
  File "/workspace/file/stable-diffusion-webui/launch.py", line 39, in main
    prepare_environment()
  File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 423, in prepare_environment
    run_pip(f"install -r \"{requirements_file}\"", "requirements")
  File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 144, in run_pip
    return run(f'"{python}" -m pip {command} --prefer-binary{index_url_line}', desc=f"Installing {desc}", errdesc=f"Couldn't install {desc}", live=live)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 116, in run
    raise RuntimeError("\n".join(error_bits))
RuntimeError: Couldn't install requirements.
Command: "/home/gitpod/.pyenv/versions/3.12.4/bin/python" -m pip install -r "requirements_versions.txt" --prefer-binary
Error code: 1

Additional information

No response

Swinec commented 3 months ago

+1, Gentoo linux, analog issue, installation step fail

Compiling tokenizers v0.13.3 (/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/tokenizers-lib)
           Running `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=89b09084cb326b58 -C extra-filename=-89b09084cb326b58 --out-dir /tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps -C strip=debuginfo -L dependency=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps --extern aho_corasick=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libaho_corasick-8ba363174647299e.rmeta --extern cached_path=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libcached_path-6bfc0561b4dbd446.rmeta --extern clap=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libclap-540591df4bacc00c.rmeta --extern derive_builder=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libderive_builder-24b0e15fc888a13f.rmeta --extern dirs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libdirs-9b2b450d525477fc.rmeta --extern esaxx_rs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libesaxx_rs-e2b589202c958bcf.rmeta --extern getrandom=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libgetrandom-0840ff858fccc57a.rmeta --extern indicatif=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libindicatif-387c5d1912c4bf6a.rmeta --extern itertools=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libitertools-244b09eccdfc1a09.rmeta --extern lazy_static=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblazy_static-70f2c43a9ded1614.rmeta --extern log=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblog-868b1533bc35336a.rmeta --extern macro_rules_attribute=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmacro_rules_attribute-7457d1e82e5afc40.rmeta --extern monostate=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmonostate-02a71e617c2c010f.rmeta --extern onig=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libonig-32f51c4c1e73388c.rmeta --extern paste=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libpaste-e36699270d1a400e.so --extern rand=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librand-c5a2a4aaba2fadc7.rmeta --extern rayon=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon-d407a8180c10eeb6.rmeta --extern rayon_cond=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon_cond-095b0ed7b7312de4.rmeta --extern regex=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex-079ed91e94dc338a.rmeta --extern regex_syntax=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex_syntax-c3ba2266c864a422.rmeta --extern reqwest=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libreqwest-7e6229659f846647.rmeta --extern serde=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde-7be8408fc50a07da.rmeta --extern serde_json=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde_json-af21dccbb02e514e.rmeta --extern spm_precompiled=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libspm_precompiled-3dde457deecc9f1c.rmeta --extern thiserror=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libthiserror-b1959683ffc698fa.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_normalization_alignments-f1efa90d7fe69706.rmeta --extern unicode_segmentation=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_segmentation-ad70de3121ea8944.rmeta --extern unicode_categories=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_categories-436bd1ea0d9dd645.rmeta -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/bzip2-sys-9f0cc99bf07a3f85/out/lib -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/zstd-sys-99695ae48306d6f1/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/esaxx-rs-4237ceb012f7dd82/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/onig_sys-43450e8faa843f01/out`
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:265:21
          |
      265 |                 let mut target_node = &mut best_path_ends_at[key_pos];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`
          |
          = note: `#[warn(unused_mut)]` on by default

      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:282:21
          |
      282 |                 let mut target_node = &mut best_path_ends_at[starts_at + mblen];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`

      warning: variable does not need to be mutable
         --> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
          |
      200 |     encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
          |                                                           ----^^^^^^^
          |                                                           |
          |                                                           help: remove this `mut`

      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
          |
      522 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      526 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default

      warning: `tokenizers` (lib) generated 3 warnings
      error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted

      Caused by:
        process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=89b09084cb326b58 -C extra-filename=-89b09084cb326b58 --out-dir /tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps -C strip=debuginfo -L dependency=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps --extern aho_corasick=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libaho_corasick-8ba363174647299e.rmeta --extern cached_path=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libcached_path-6bfc0561b4dbd446.rmeta --extern clap=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libclap-540591df4bacc00c.rmeta --extern derive_builder=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libderive_builder-24b0e15fc888a13f.rmeta --extern dirs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libdirs-9b2b450d525477fc.rmeta --extern esaxx_rs=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libesaxx_rs-e2b589202c958bcf.rmeta --extern getrandom=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libgetrandom-0840ff858fccc57a.rmeta --extern indicatif=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libindicatif-387c5d1912c4bf6a.rmeta --extern itertools=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libitertools-244b09eccdfc1a09.rmeta --extern lazy_static=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblazy_static-70f2c43a9ded1614.rmeta --extern log=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/liblog-868b1533bc35336a.rmeta --extern macro_rules_attribute=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmacro_rules_attribute-7457d1e82e5afc40.rmeta --extern monostate=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libmonostate-02a71e617c2c010f.rmeta --extern onig=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libonig-32f51c4c1e73388c.rmeta --extern paste=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libpaste-e36699270d1a400e.so --extern rand=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librand-c5a2a4aaba2fadc7.rmeta --extern rayon=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon-d407a8180c10eeb6.rmeta --extern rayon_cond=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/librayon_cond-095b0ed7b7312de4.rmeta --extern regex=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex-079ed91e94dc338a.rmeta --extern regex_syntax=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libregex_syntax-c3ba2266c864a422.rmeta --extern reqwest=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libreqwest-7e6229659f846647.rmeta --extern serde=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde-7be8408fc50a07da.rmeta --extern serde_json=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libserde_json-af21dccbb02e514e.rmeta --extern spm_precompiled=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libspm_precompiled-3dde457deecc9f1c.rmeta --extern thiserror=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libthiserror-b1959683ffc698fa.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_normalization_alignments-f1efa90d7fe69706.rmeta --extern unicode_segmentation=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_segmentation-ad70de3121ea8944.rmeta --extern unicode_categories=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/deps/libunicode_categories-436bd1ea0d9dd645.rmeta -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/bzip2-sys-9f0cc99bf07a3f85/out/lib -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/zstd-sys-99695ae48306d6f1/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/esaxx-rs-4237ceb012f7dd82/out -L native=/tmp/pip-install-lfnqsm52/tokenizers_4a141a2d2c4a4feca24370febaf9c32b/target/release/build/onig_sys-43450e8faa843f01/out` (exit status: 1)
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

FournyP commented 2 months ago

I face the same issue while installing requirements :

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

xangelix commented 2 months ago

This issue "shouldn't" happen if you are using python 3.10, the highest supported version for this project. As much as I wish we had a more up to date version of tokenizers and python compatible-- we don't.

Perhaps this is caused because pypi does not have prebuilt binaries for that old of a version of tokenizers for that new of a version of python?

Repost here if python 3.10.x doesn't fix it!

projecthorizon993 commented 2 months ago

nope still 3.10.12 still have that bug idk what todo still tokenizer issue

mmmpipi commented 2 months ago

same problem on ArchLinux

projecthorizon993 commented 2 months ago

same problem on ArchLinux

fixed the tokenizer

this one mental abuse me for over a week

Commit hash: 82a973c04367123ae98bd9abdf80d9eda9b910e2 Launching Web UI with arguments: --listen --theme dark --skip-torch-cuda-test --no-half --use-cpu all --share Traceback (most recent call last): File "/workspace/file/stable-diffusion-webui/launch.py", line 48, in main() File "/workspace/file/stable-diffusion-webui/launch.py", line 44, in main start() File "/workspace/file/stable-diffusion-webui/modules/launch_utils.py", line 465, in start import webui File "/workspace/file/stable-diffusion-webui/webui.py", line 13, in initialize.imports() File "/workspace/file/stable-diffusion-webui/modules/initialize.py", line 17, in imports import pytorch_lightning # noqa: F401 File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/init.py", line 35, in from pytorch_lightning.callbacks import Callback # noqa: E402 File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/callbacks/init.py", line 14, in from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in from pytorch_lightning.callbacks.callback import Callback File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in from pytorch_lightning.utilities.types import STEP_OUTPUT File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 27, in from torchmetrics import Metric ... rcParams['backend'] = os.environ.get('MPLBACKEND') File "/home/gitpod/.pyenv/versions/3.10.6/lib/python3.10/site-packages/matplotlib/init.py", line 738, in setitem raise ValueError(f"Key {key}: {ve}") from None ValueError: Key backend: 'module://matplotlib_inline.backend_inline' is not a valid value for backend; supported values are ['gtk3agg', 'gtk3cairo', 'gtk4agg', 'gtk4cairo', 'macosx', 'nbagg', 'notebook', 'qtagg', 'qtcairo', 'qt5agg', 'qt5cairo', 'tkagg', 'tkcairo', 'webagg', 'wx', 'wxagg', 'wxcairo', 'agg', 'cairo', 'pdf', 'pgf', 'ps', 'svg', 'template']

fishBone000 commented 2 months ago

This could be caused by incorrect rustc version?

I have issue with toenizers as well on my Arch Linux, in my error log it hints I should compile it with rustc 1.74.0 or later, but I'm using 1.70, which is installed by rustup package.
We have the same exit code (101), but I can't find similar rustc message in your error logs.

I'm trying to fix my own issue.

projecthorizon993 commented 2 months ago

This could be caused by incorrect rustc version?

I have issue with toenizers as well on my Arch Linux, in my error log it hints I should compile it with rustc 1.74.0 or later, but I'm using 1.70, which is installed by rustup package. We have the same exit code (101), but I can't find similar rustc message in your error logs.

I'm trying to fix my own issue.

could be but I think it more likely a bug on the tokenizer code that complie with older rust so in newer ver it just broke maybe

projecthorizon993 commented 2 months ago

This could be caused by incorrect rustc version?

I have issue with toenizers as well on my Arch Linux, in my error log it hints I should compile it with rustc 1.74.0 or later, but I'm using 1.70, which is installed by rustup package. We have the same exit code (101), but I can't find similar rustc message in your error logs.

I'm trying to fix my own issue.

I found out 1.65 work maybe but it break when I reload new model

mikeesouth commented 1 month ago

Getting the same problem on my M1 Mac. Tried downgrading to rust 1.65 and python 3.10.12

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

Any workarounds?

wangyang0210 commented 1 month ago

some problem on my M2 Max Mac, I slove this question use conda create env use python 3.10.15.

ledyba commented 1 month ago

It looks original sources are already fixed (with unsafe keyword)

https://github.com/huggingface/tokenizers/blob/6ea758872d74a0f1c4cd5b6a56e0fdd4f3a10418/tokenizers/src/models/bpe/trainer.rs#L540-L547

So... is there a update of python libs? (I can't find what library uses this crate.)

Lolihunter1337 commented 1 month ago

Stuck with the same issue

projecthorizon993 commented 1 month ago

Getting the same problem on my M1 Mac. Tried downgrading to rust 1.65 and python 3.10.12

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

Any workarounds?

M1 I think you need to have more packet btw that more likely a bug of ARM emulation

projecthorizon993 commented 1 month ago

It looks original sources are already fixed (with unsafe keyword)

https://github.com/huggingface/tokenizers/blob/6ea758872d74a0f1c4cd5b6a56e0fdd4f3a10418/tokenizers/src/models/bpe/trainer.rs#L540-L547

So... is there a update of python libs? (I can't find what library uses this crate.)

no it just a really bug from python itself idk but I solve this using a older version of rust and ignore that bug

ledyba commented 1 month ago

I'm using this condition:

Python 3.12.3
Rustc 1.82.0 (f6e511eec 2024-10-15)
AMD ROCm (RTX6800)

I tried rustc 1.65.0, but it does not solve the problem.

Finally, I rewrite requirements_versions.txt and change transformers version to 4.34.0 (from 4.30.2).

transformers==4.34.0

Thus, I think it is neither a bug of python or rustc.

ledyba commented 1 month ago

Sorry, also you need this patch, because of the update of transformer:

https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/13245#issuecomment-1766452012

jstelzer commented 2 days ago

Hit this issue when I accidentally used the wrong python version myself.

you absolutely need to be on 3.10.6.

if your build is failing due to unsafe rust code, then you likely are using the wrong python runtime. But the issue you're hitting is actually over here: https://github.com/huggingface/tokenizers/issues/1485

Time continues to move forward. Libraries that the dependencies in requirements.txt depend on release new versions. Pip wanders into them. If you were using poetry instead of bare pip, then at least you could check in the lock file with the sum total of deps and their deps which would probably make for a more reproducible build. But when you mix and match major python versions, all bets are off.

pyenv to the rescue.

You know what? Running this out of a docker image would be pretty straight forward and might cut down on the finicky python env sensitivity. Just a thought.

AUTOMATIC1111 / stable-diffusion-webui