compilation error during installation via pip

[Darskiy]

      error[E0425]: cannot find function, tuple struct or tuple variant `Some` in this scope
         --> C:\Users\SerjD\.cargo\registry\src\index.crates.io-6f17d22bba15001f\regex-syntax-0.7.4\src\utf8.rs:398:13
          |
      398 |             Some((
          |             ^^^^ not found in this scope

      error[E0425]: cannot find function, tuple struct or tuple variant `Some` in this scope
         --> C:\Users\SerjD\.cargo\registry\src\index.crates.io-6f17d22bba15001f\regex-syntax-0.7.4\src\utf8.rs:418:13
          |
      418 |             Some(Utf8Range::new(start, end))
          |             ^^^^ not found in this scope

      Some errors have detailed explanations: E0405, E0408, E0412, E0416, E0425, E0463, E0531.
      error: could not compile `regex-syntax` (lib) due to 1299 previous errors

      Caused by:
        process didn't exit successfully: `rustc --crate-name regex_syntax --edition=2021 C:\Users\SerjD\.cargo\registry\src\index.crates.io-6f17d22bba15001f\regex-syntax-0.7.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" --cfg "feature=\"unicode\"" --cfg "feature=\"unicode-age\"" --cfg "feature=\"unicode-bool\"" --cfg "feature=\"unicode-case\"" --cfg "feature=\"unicode-gencat\"" --cfg "feature=\"unicode-perl\"" --cfg "feature=\"unicode-script\"" --cfg "feature=\"unicode-segment\"" -C metadata=fbbc80bfc1eb39ee -C extra-filename=-fbbc80bfc1eb39ee --out-dir C:\Users\SerjD\AppData\Local\Temp\pip-install-baz_bpnf\tiktoken_6c12c2ab3f82450693b000c96abb76d0\target\i686-pc-windows-msvc\release\deps --target i686-pc-windows-msvc -L dependency=C:\Users\SerjD\AppData\Local\Temp\pip-install-baz_bpnf\tiktoken_6c12c2ab3f82450693b000c96abb76d0\target\i686-pc-windows-msvc\release\deps -L dependency=C:\Users\SerjD\AppData\Local\Temp\pip-install-baz_bpnf\tiktoken_6c12c2ab3f82450693b000c96abb76d0\target\release\deps --cap-lints allow` (exit code: 1)
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --target i686-pc-windows-msvc --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tiktoken
Failed to build tiktoken
ERROR: Could not build wheels for tiktoken, which is required to install pyproject.toml-based projects

pip == 23.2.1 wheel == 0.41.1 rustc 1.71.1 (eb26296b5 2023-08-03)

[hauntsaninja]

That's a pretty weird error. Anything special about your rust setup?

[Darskiy]

That's a pretty weird error. Anything special about your rust setup?

Nothing specific, downloaded the 32bit installer from the official site -> installed, that's all. The compiler itself works properly, at least it copes with helloworld. OS - windows 11 if that is important

[hauntsaninja]

Sorry, I'm not sure what's wrong with your setup. Closing since it's not really linked to tiktoken — the failure is in compiling the Rust dependencies for tiktoken. Seems like it's literally failing to find Some. I'd definitely try uninstalling all copies of Rust on your system and reinstalling if you haven't already.

[Darskiy]

Just tried to install on a fresh system in vmware and ran into the same problem :( Only python 3.11.4 and the rustc compiler with all the necessary c++ library dependencies are installed

The systems on which it was tested: Windows 11 Education\Pro (build 22621.2134) Windows 10 IoT Enterprise (build 19043.1021)

[Darskiy]

@hauntsaninja The problem was solved by updating the interpreter from the 32-bit to the 64-bit version. Apparently, tiktoken does not support 32-bit wheels