search

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
9.08k stars 807 forks source link

compile error when installing versions 0.9.2 or 0.8.1.rc2 #874

Closed bth5032 closed 8 months ago

bth5032 commented 2 years ago

I ran into this issue when installing transformers[flax] which seems to depend on older versions of tokenizers. I do have the latest tokenizers available installed via pip, but attempting to install tokenizers==0.9.2 gives some rust compilation error:

Collecting tokenizers==0.8.1.rc2
  Using cached tokenizers-0.8.1rc2.tar.gz (97 kB)
  Installing build dependencies ... done
  ...
  copying tokenizers\processors\__init__.pyi -> build\lib.win-amd64-3.9\tokenizers\processors
  copying tokenizers\trainers\__init__.pyi -> build\lib.win-amd64-3.9\tokenizers\trainers
  running build_ext
      Updating crates.io index
      Updating git repository `https://github.com/n1t0/rayon-cond`
  cargo rustc --lib --manifest-path Cargo.toml --target x86_64-pc-windows-msvc --release -v --features pyo3/extension-module -- --crate-type cdylib
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.36
     Compiling unicode-xid v0.2.2
     Compiling syn v1.0.85
     Compiling autocfg v1.0.1
     Compiling memchr v2.4.1
     Compiling serde_derive v1.0.133
     Compiling cfg-if v1.0.0
     Compiling serde v1.0.133
       Running `rustc --crate-name build_script_build --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.36\build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=57eed64b2791e1b0 -C extra-filename=-57eed64b2791e1b0 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\build\proc-macro2-57eed64b2791e1b0 -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --cap-lints allow`
       Running `rustc --crate-name unicode_xid C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-xid-0.2.2\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts 
--crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"default\"" -C metadata=6f559fb7bfe8c6bc -C extra-filename=-6f559fb7bfe8c6bc --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.85\build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg 
"feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=1244fbe766a9b584 -C extra-filename=-1244fbe766a9b584 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\build\syn-1244fbe766a9b584 -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --cap-lints allow`
       Running `rustc --crate-name autocfg C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\autocfg-1.0.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=8f32ca860fd83420 -C extra-filename=-8f32ca860fd83420 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --cap-lints 
allow`
       Running `rustc --crate-name build_script_build --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.4.1\build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=d0f507365936f9e9 -C extra-filename=-d0f507365936f9e9 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\build\memchr-d0f507365936f9e9 -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --cap-lints allow`
...
     Compiling paste v0.1.18
       Running `rustc --crate-name paste --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\paste-0.1.18\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=d8f761c94ebe3ef5 -C extra-filename=-d8f761c94ebe3ef5 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --extern paste_impl=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps\paste_impl-492a7b1b1e07bb82.dll --extern proc_macro_hack=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps\proc_macro_hack-2d55d09683b3196f.dll --cap-lints allow`
     Compiling console v0.15.0
       Running `rustc --crate-name console --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\console-0.15.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"ansi-parsing\"" --cfg "feature=\"default\"" --cfg "feature=\"regex\"" --cfg "feature=\"unicode-width\"" -C metadata=7f04b326976a8469 -C extra-filename=-7f04b326976a8469 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --extern encode_unicode=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libencode_unicode-f5082f2d76afde34.rmeta --extern libc=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\liblibc-3099232219a69ddb.rmeta --extern once_cell=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libonce_cell-e26956b93b1a289e.rmeta --extern regex=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libregex-209ee3cef779bb9d.rmeta --extern terminal_size=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libterminal_size-caf7104c0642fe58.rmeta --extern unicode_width=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libunicode_width-ce66de2e14cc3ba9.rmeta --extern winapi=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libwinapi-3c77d9c468bf0363.rmeta --cap-lints allow`
     Compiling clap v2.34.0
       Running `rustc --crate-name clap --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\clap-2.34.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"ansi_term\"" --cfg "feature=\"atty\"" --cfg "feature=\"color\"" --cfg "feature=\"default\"" --cfg "feature=\"strsim\"" --cfg "feature=\"suggestions\"" --cfg "feature=\"vec_map\"" -C metadata=7b7912e462d4241a -C extra-filename=-7b7912e462d4241a --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --extern atty=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libatty-0152d3558cda1980.rmeta --extern bitflags=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libbitflags-1d434c93db9ceb5c.rmeta --extern strsim=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libstrsim-563e93c49ab77cd6.rmeta --extern textwrap=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libtextwrap-d44db99661ee5bc3.rmeta --extern unicode_width=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libunicode_width-ce66de2e14cc3ba9.rmeta --extern vec_map=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libvec_map-f82cb06a00d96a3c.rmeta --cap-lints allow`
  error[E0658]: `if` is not allowed in a `const fn`
    --> C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\clap-2.34.0\src\app\settings.rs:7:1
     |
  7  | / bitflags! {
  8  | |     struct Flags: u64 {
  9  | |         const SC_NEGATE_REQS       = 1;
  10 | |         const SC_REQUIRED          = 1 << 1;
  ...  |
  51 | |     }
  52 | | }
     | |_^
     |
     = note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
     = help: add `#![feature(const_if_match)]` to the crate attributes to enable
     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

  error[E0658]: `if` is not allowed in a `const fn`
    --> C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\clap-2.34.0\src\args\settings.rs:6:1
     |
  6  | / bitflags! {
  7  | |     struct Flags: u32 {
  8  | |         const REQUIRED         = 1;
  9  | |         const MULTIPLE         = 1 << 1;
  ...  |
  28 | |     }
  29 | | }
     | |_^
     |
     = note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
     = help: add `#![feature(const_if_match)]` to the crate attributes to enable
     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

     Compiling parking_lot v0.10.2
       Running `rustc --crate-name parking_lot --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\parking_lot-0.10.2\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"nightly\"" -C metadata=f2e6ca23d2714e74 -C extra-filename=-f2e6ca23d2714e74 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc 
-L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --extern lock_api=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\liblock_api-feeea9924ab45186.rmeta --extern parking_lot_core=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a21     Compiling crossbeam-deque v0.8.1
       Running `rustc --crate-name crossbeam_deque --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-deque-0.8.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"crossbeam-epoch\"" --cfg "feature=\"crossbeam-utils\"" --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=f5042eb50925de89 -C extra-filename=-f5042eb50925de89 --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --extern cfg_if=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libcfg_if-cbe2e508f7dd2e0d.rmeta --extern crossbeam_epoch=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libcrossbeam_epoch-a64cc1bf8654937d.rmeta --extern crossbeam_utils=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libcrossbeam_utils-01e26c75846d7d46.rmeta --cap-lints allow`
     Compiling indicatif v0.14.0
       Running `rustc --crate-name indicatif --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\indicatif-0.14.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"default\"" -C metadata=4405e1307bee3b7a -C extra-filename=-4405e1307bee3b7a --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --extern console=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libconsole-7f04b326976a8469.rmeta --extern lazy_static=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\liblazy_static-38c6c694dbdce5e7.rmeta --extern number_prefix=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libnumber_prefix-e34de271db8483ce.rmeta --extern regex=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libregex-209ee3cef779bb9d.rmeta --cap-lints allow`
  error: aborting due to 2 previous errors

  For more information about this error, try `rustc --explain E0658`.
  error: could not compile `clap`.

  Caused by:
    process didn't exit successfully: `rustc --crate-name clap --edition=2018 C:\Users\Bobak\.cargo\registry\src\github.com-1ecc6299db9ec823\clap-2.34.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg "feature=\"ansi_term\"" --cfg "feature=\"atty\"" --cfg "feature=\"color\"" --cfg "feature=\"default\"" --cfg "feature=\"strsim\"" --cfg "feature=\"suggestions\"" --cfg "feature=\"vec_map\"" -C metadata=7b7912e462d4241a -C extra-filename=-7b7912e462d4241a --out-dir C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\release\deps --extern atty=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libatty-0152d3558cda1980.rmeta --extern bitflags=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libbitflags-1d434c93db9ceb5c.rmeta --extern strsim=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libstrsim-563e93c49ab77cd6.rmeta --extern textwrap=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libtextwrap-d44db99661ee5bc3.rmeta 
--extern unicode_width=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libunicode_width-ce66de2e14cc3ba9.rmeta --extern vec_map=C:\Users\Bobak\AppData\Local\Temp\pip-install-ze8nr3yc\tokenizers_190fd1436b984ea5a3bad2d8a2102e79\target\x86_64-pc-windows-msvc\release\deps\libvec_map-f82cb06a00d96a3c.rmeta --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
  error: build failed
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Can someone help me figure out what's going on? This is on windows 10 running python 3.9.1 and rustc 1.57.0

Narsil commented 2 years ago

Hmm this seems pretty odd.

Version 0.8.1rc2 does seems to have a bug in it, however it is an extremely old version, it should never have to go back there. Which version of pip are you using ? It could be the somewhat old version that kept attempting every version until it resolved all dependencies.

Can you try forcing the version <0.11 with pip install tokenizers==0.10.3 before pip install transformers[flax] that might solve the issue.

I don't have a windows specifically to test, but versions >0.10 (at least) should all work on windows. I am going to try and reproduce the full behavior too.

bth5032 commented 2 years ago

Hi @Narsil , it seems I already have 0.10.3 as pip says requirement is already satisfied (I guess this is the latest version?). Here is my pip freeze

absl-py==0.14.0
aiohttp==3.8.1
aiosignal==1.2.0
anyio==2.1.0
appdirs==1.4.4
argon2-cffi==20.1.0
astunparse==1.6.3
async-generator==1.10
async-timeout==4.0.2
atomicwrites==1.4.0
attrs==20.3.0
Babel==2.9.0
backcall==0.2.0
base58==2.1.1
bidict==0.21.2
bitarray==1.2.2
bleach==3.3.0
blobfile==1.2.5
boto3==1.18.48
botocore==1.21.48
cachetools==4.2.2
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
charset-normalizer==2.0.9
clang==5.0
click==7.1.2
click-default-group==1.2.2
clip==1.0
cloup==0.7.1
colorama==0.4.4
colour==0.1.5
commonmark==0.9.1
cycler==0.10.0
cytoolz==0.11.2
DALL-E==0.1
decorator==5.0.9
defusedxml==0.6.0
distlib==0.3.1
dnspython==1.16.0
entrypoints==0.3
eth-abi==2.1.1
eth-account==0.5.6
eth-hash==0.3.2
eth-keyfile==0.5.1
eth-keys==0.3.3
eth-rlp==0.2.1
eth-typing==2.2.2
eth-utils==1.10.0
eventlet==0.30.2
filelock==3.0.12
Flask==1.1.2
Flask-SocketIO==5.0.1
flatbuffers==1.12
flax==0.3.3
frozenlist==1.2.0
ftfy==6.0.3
gast==0.4.0
glcontext==2.3.4
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
greenlet==1.0.0
grpcio==1.40.0
h5py==3.1.0
hexbytes==0.2.2
huggingface-hub==0.2.1
idna==2.10
iexfinance==0.5.0
imageio==2.9.0
iniconfig==1.1.1
ipfshttpclient==0.8.0a2
ipykernel==5.4.3
ipython==7.20.0
ipython-genutils==0.2.0
itsdangerous==1.1.0
jax==0.2.26
jedi==0.18.0
Jinja2==2.11.3
jmespath==0.10.0
joblib==1.0.1
json5==0.9.5
jsonschema==3.2.0
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyter-server==1.3.0
jupyterlab==3.0.7
jupyterlab-hdf==0.6.0
jupyterlab-pygments==0.1.2
jupyterlab-server==2.2.0
keras==2.6.0
Keras-Preprocessing==1.1.2
kiwisolver==1.3.1
kornia==0.5.11
lru-dict==1.1.7
manim==0.9.0
ManimPango==0.3.0
mapbox-earcut==0.12.10
Markdown==3.3.4
MarkupSafe==1.1.1
matplotlib==3.3.4
mistune==0.8.4
moderngl==5.6.4
moderngl-window==2.4.0
mpmath==1.2.1
msgpack==1.0.3
multiaddr==0.0.9
multidict==5.2.0
multipledispatch==0.6.0
mypy==0.910
mypy-extensions==0.4.3
nbclassic==0.2.6
nbclient==0.5.2
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
netaddr==0.8.0
networkx==2.6.2
notebook==6.2.0
numpy==1.19.5
oauthlib==3.1.1
opencv-python==4.5.2.52
opt-einsum==3.3.0
packaging==20.9
pandas==1.2.4
pandoc==1.1.0
pandocfilters==1.4.3
parsimonious==0.8.1
parso==0.8.1
pickleshare==0.7.5
Pillow==8.1.0
pipenv==2020.11.15
plotly==5.3.1
pluggy==1.0.0
plumbum==1.7.0
ply==3.11
praw==7.2.0
prawcore==2.0.0
prometheus-client==0.9.0
prompt-toolkit==3.0.16
protobuf==3.18.0
py==1.10.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.20.1
pycparser==2.20
pycryptodome==3.12.0
pycryptodomex==3.10.4
pydub==0.25.1
pygame==2.0.1
pyglet==1.5.18
Pygments==2.7.4
PyOpenGL==3.1.5
pyparsing==2.4.7
pypiwin32==223
PyQt5==5.15.6
PyQt5-Qt5==5.15.2
PyQt5-sip==12.9.0
pyrr==0.10.3
pyrsistent==0.17.3
PySide2==5.15.2
pytest==6.2.5
python-dateutil==2.8.1
python-engineio==4.0.0
python-socketio==5.0.4
pytz==2021.1
pywin32==300
pywinpty==0.5.7
PyYAML==5.4.1
pyzmq==22.0.3
regex==2021.9.24
requests==2.25.1
requests-oauthlib==1.3.0
rich==10.7.0
rlp==2.0.1
rsa==4.7.2
s3transfer==0.5.0
sacremoses==0.0.46
scikit-learn==0.24.1
scipy==1.6.0
screeninfo==0.6.7
Send2Trash==1.5.0
shiboken2==5.15.2
simplejson==3.17.5
six==1.15.0
sklearn==0.0
sniffio==1.2.0
sympy==1.7.1
tenacity==8.0.1
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.6.0
tensorflow-estimator==2.6.0
termcolor==1.1.0
terminado==0.9.2
testpath==0.4.4
threadpoolctl==2.1.0
tokenizers==0.10.3
toml==0.10.2
toolz==0.11.2
torch==1.9.1+cu102
torchaudio==0.9.1
torchvision==0.10.1+cu102
tornado==6.1
tqdm==4.62.1
traitlets==5.0.5
transformers==4.15.0
typing-extensions==3.7.4.3
update-checker==0.18.0
urllib3==1.26.3
varint==1.0.2
virtualenv==20.4.4
virtualenv-clone==0.5.4
visvis==1.13.0
watchdog==2.1.3
wcwidth==0.2.5
web3==5.25.0
webencodings==0.5.1
websocket-client==0.58.0
websockets==9.1
Werkzeug==1.0.1
wget==3.2
wrapt==1.12.1
wxPython==4.1.1
xmltodict==0.12.0
yarl==1.7.2
bth5032 commented 2 years ago

Could this line have something to do with it

Narsil commented 2 years ago

possibly. Is jax now supported on windows ? If yes, you can install pip install transformers and pip install jax separately, and everything should work.

If that's the case, you may have found the culprit (don't know how empty lists end up working but maybe it messes things up ?)

kaaquist commented 2 years ago

I have the similar error. I know I'm installing a old version of transformers==3.3.1 (same used for [flax]) which relay on tokenizers between 0.9.2 and 0.8.1.rc2. It use to work in my CI/CD pipeline but with the latest version of cargo==1.57.0 it does not work any more.

(not using Windows) (Local machine)

> uname -v
#102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021

(docker image)

> docker run --entrypoint "/bin/bash" -it python:3.9.9-buster
root@b87c44cc4109:/# uname -v
#102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021
root@b87c44cc4109:/# python --version
Python 3.9.9
root@b87c44cc4109:/#

I get the same compile error:

--cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c8c722b05cb19780 -C extra-filename=-c8c722b05cb19780 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow --cfg has_i128 --cfg has_to_int_unchecked`
     Compiling rand_core v0.5.1
       Running `rustc --crate-name rand_core --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="getrandom"' --cfg 'feature="std"' -C metadata=41b47b15dbcf2b50 -C extra-filename=-41b47b15dbcf2b50 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern getrandom=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libgetrandom-22bce8e58e48dc75.rmeta --cap-lints allow`
     Compiling clap v2.34.0
       Running `rustc --crate-name clap --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=d1d931e823cd42c4 -C extra-filename=-d1d931e823cd42c4 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern ansi_term=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libansi_term-299089b8a23df6cc.rmeta --extern atty=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libatty-601625c331c876fb.rmeta --extern bitflags=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libbitflags-07e4b438a47db117.rmeta --extern strsim=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libstrsim-0b9af79a9eabd5cc.rmeta --extern textwrap=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libtextwrap-1e5358ff7f49d80c.rmeta --extern unicode_width=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --extern vec_map=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libvec_map-99222fb3abba5a69.rmeta --cap-lints allow`
     Compiling parking_lot v0.10.2
       Running `rustc --crate-name parking_lot --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.10.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="nightly"' -C metadata=c06e3469bd781de6 -C extra-filename=-c06e3469bd781de6 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern lock_api=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblock_api-0e8ba3e8e8af44f0.rmeta --extern parking_lot_core=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libparking_lot_core-6839db6955175ef9.rmeta --cap-lints allow`
  error[E0658]: `if` is not allowed in a `const fn`
    --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/app/settings.rs:7:1
     |
  7  | / bitflags! {
  8  | |     struct Flags: u64 {
  9  | |         const SC_NEGATE_REQS       = 1;
  10 | |         const SC_REQUIRED          = 1 << 1;
  ...  |
  51 | |     }
  52 | | }
     | |_^
     |
     = note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
     = help: add `#![feature(const_if_match)]` to the crate attributes to enable
     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

  error[E0658]: `if` is not allowed in a `const fn`
    --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/args/settings.rs:6:1
     |
  6  | / bitflags! {
  7  | |     struct Flags: u32 {
  8  | |         const REQUIRED         = 1;
  9  | |         const MULTIPLE         = 1 << 1;
  ...  |
  28 | |     }
  29 | | }
     | |_^
     |
     = note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
     = help: add `#![feature(const_if_match)]` to the crate attributes to enable
     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

     Compiling regex v1.5.4
       Running `rustc --crate-name regex --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.5.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=1e333a3fef530f25 -C extra-filename=-1e333a3fef530f25 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern aho_corasick=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libaho_corasick-157f3d95428c1664.rmeta --extern memchr=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libmemchr-671cb2e32aa232f7.rmeta --extern regex_syntax=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libregex_syntax-79e1cae865a51a95.rmeta --cap-lints allow`
     Compiling crossbeam-deque v0.8.1
       Running `rustc --crate-name crossbeam_deque --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="crossbeam-epoch"' --cfg 'feature="crossbeam-utils"' --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=98dbc32454e4e8b2 -C extra-filename=-98dbc32454e4e8b2 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern cfg_if=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern crossbeam_epoch=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_epoch-d30c0882f3ed7f6d.rmeta --extern crossbeam_utils=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_utils-45ceb0003524c71c.rmeta --cap-lints allow`
     Compiling rand_chacha v0.2.2
       Running `rustc --crate-name rand_chacha --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_chacha-0.2.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=e7ad19857ddbc1d9 -C extra-filename=-e7ad19857ddbc1d9 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern ppv_lite86=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libppv_lite86-f41c402efdf936bd.rmeta --extern rand_core=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/librand_core-41b47b15dbcf2b50.rmeta --cap-lints allow`
       Running `rustc --crate-name rayon_core --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=af1f8b9b5f2ff68e -C extra-filename=-af1f8b9b5f2ff68e --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern crossbeam_channel=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_channel-79a318de3ba8ac08.rmeta --extern crossbeam_deque=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_deque-98dbc32454e4e8b2.rmeta --extern crossbeam_utils=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_utils-45ceb0003524c71c.rmeta --extern lazy_static=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblazy_static-79cb8478fcd4b466.rmeta --extern num_cpus=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libnum_cpus-d1eb7316a17ef3b0.rmeta --cap-lints allow`
     Compiling paste-impl v0.1.18
       Running `rustc --crate-name paste_impl --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -Cembed-bitcode=no -C metadata=7047fcfa649dd5ba -C extra-filename=-7047fcfa649dd5ba --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern proc_macro_hack=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libproc_macro_hack-940fbea956d42b7d.so --extern proc_macro --cap-lints allow`
  error: aborting due to 2 previous errors

  For more information about this error, try `rustc --explain E0658`.
     Compiling rand v0.7.3
  error: could not compile `clap`.

  Caused by:
    process didn't exit successfully: `rustc --crate-name clap --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=d1d931e823cd42c4 -C extra-filename=-d1d931e823cd42c4 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern ansi_term=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libansi_term-299089b8a23df6cc.rmeta --extern atty=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libatty-601625c331c876fb.rmeta --extern bitflags=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libbitflags-07e4b438a47db117.rmeta --extern strsim=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libstrsim-0b9af79a9eabd5cc.rmeta --extern textwrap=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libtextwrap-1e5358ff7f49d80c.rmeta --extern unicode_width=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --extern vec_map=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libvec_map-99222fb3abba5a69.rmeta --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
       Running `rustc --crate-name rand --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="getrandom"' --cfg 'feature="getrandom_package"' --cfg 'feature="libc"' --cfg 'feature="std"' -C metadata=b6e3f777bcce4393 -C extra-filename=-b6e3f777bcce4393 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern getrandom_package=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libgetrandom-22bce8e58e48dc75.rmeta --extern libc=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblibc-d1d545bbab16b389.rmeta --extern rand_chacha=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/librand_chacha-e7ad19857ddbc1d9.rmeta --extern rand_core=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/librand_core-41b47b15dbcf2b50.rmeta --cap-lints allow`
  error: build failed
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
The command '/bin/sh -c . $HOME/.cargo/env && pip install -r requirements-docker.txt' returned a non-zero code: 1
>

I would like to avoid to bump the version of transtormer since ver. 4.5.0 is not backwards compatible with the version I have now and that would mean I need to do a lot to get my pipeline running again. So is there any work around to get this here running again. Log from when it use to work (Started on Jun 25, 2021, 1:42:05 PM):

Step #0 - "Build Docker Image": info: default host triple is x86_64-unknown-linux-gnu
Step #0 - "Build Docker Image": info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
Step #0 - "Build Docker Image": info: latest update on 2021-06-17, rust version 1.53.0 (53cb7b09b 2021-06-17)
Step #0 - "Build Docker Image": info: downloading component 'cargo'
Step #0 - "Build Docker Image": info: downloading component 'clippy'
Step #0 - "Build Docker Image": info: downloading component 'rust-docs'
Step #0 - "Build Docker Image": info: downloading component 'rust-std'
Step #0 - "Build Docker Image": info: downloading component 'rustc'
Step #0 - "Build Docker Image": info: downloading component 'rustfmt'
Step #0 - "Build Docker Image": info: installing component 'cargo'
Step #0 - "Build Docker Image": info: installing component 'clippy'
Step #0 - "Build Docker Image": info: installing component 'rust-docs'
Step #0 - "Build Docker Image": info: installing component 'rust-std'
Step #0 - "Build Docker Image": info: installing component 'rustc'
Step #0 - "Build Docker Image": info: installing component 'rustfmt'
Step #0 - "Build Docker Image": 
Step #0 - "Build Docker Image": info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
Step #0 - "Build Docker Image":   stable-x86_64-unknown-linux-gnu installed - rustc 1.53.0 (53cb7b09b 2021-06-17)
Step #0 - "Build Docker Image": 
Step #0 - "Build Docker Image": 
Step #0 - "Build Docker Image": Rust is installed now. Great!
Step #0 - "Build Docker Image": 
Step #0 - "Build Docker Image": To get started you may need to restart your current shell.
Step #0 - "Build Docker Image": This would reload your PATH environment variable to include
Step #0 - "Build Docker Image": Cargo's bin directory ($HOME/.cargo/bin).
Step #0 - "Build Docker Image": 
Step #0 - "Build Docker Image": To configure your current shell, run:
...
...
...
Step #0 - "Build Docker Image": Building wheels for collected packages: tokenizers
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): started
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): still running...
Step #0 - "Build Docker Image":   Building wheel for tokenizers (PEP 517): finished with status 'done'
Step #0 - "Build Docker Image":   Created wheel for tokenizers: filename=tokenizers-0.8.1rc2-cp39-cp39-linux_x86_64.whl size=3064027 sha256=355a99836d8b871e535c5f7b603c78b6abaca57511122a8e99de283fd685ab1d
Step #0 - "Build Docker Image":   Stored in directory: /root/.cache/pip/wheels/4e/30/81/b87bbe7da11c6eb57d6d24e8cfb386615d0c8e7f76a70e6b79
Step #0 - "Build Docker Image": Successfully built tokenizers
...
...
kaaquist commented 2 years ago

Ok, it seems that it might be related to this here: 

...
copying tokenizers/trainers/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers/trainers
  running build_ext
  info: syncing channel updates for 'nightly-2020-05-14-x86_64-unknown-linux-gnu'
  info: latest update on 2020-05-14, rust version 1.45.0-nightly (75e1463c5 2020-05-13)
  info: downloading component 'cargo'
...

It seems like i changes the rust vertion to 1.45.0 when building the tokenizer.

Full log:

> docker build -t stcs-demo -f ./docker/Dockerfile .
Sending build context to Docker daemon  111.1kB
Step 1/16 : FROM python:3.9.9-buster
 ---> 9d51e487eb7b
Step 2/16 : ARG gitsha=xxxx
 ---> Using cache
 ---> 514aee253cde
Step 3/16 : ENV GITSHALABEL="${gitsha}"
 ---> Using cache
 ---> a7c61ba5feff
Step 4/16 : LABEL version="${GITSHALABEL}"
 ---> Using cache
 ---> 07c5c0b666d7
Step 5/16 : RUN apt-get update
 ---> Using cache
 ---> 0bd2cd04c85a
Step 6/16 : RUN pip install --upgrade pip setuptools wheel
 ---> Using cache
 ---> aa4339d95ec7
Step 7/16 : WORKDIR /opt/app/
 ---> Using cache
 ---> fea21f746cf1
Step 8/16 : COPY requirements-docker.txt /opt/app
 ---> Using cache
 ---> 613079f65e59
Step 9/16 : RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
 ---> Running in 65825e057ac5
info: downloading installer
info: profile set to 'default'
info: default host triple is x86_64-unknown-linux-gnu
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2021-12-02, rust version 1.57.0 (f1edd0429 2021-11-29)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
info: installing component 'rust-std'
info: installing component 'rustc'
info: installing component 'rustfmt'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.57.0 (f1edd0429 2021-11-29)

Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source $HOME/.cargo/env
Removing intermediate container 65825e057ac5
 ---> 844664b88340
Step 10/16 : RUN . $HOME/.cargo/env && rustup update
 ---> Running in b52352efa9e0
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: checking for self-updates

  stable-x86_64-unknown-linux-gnu unchanged - rustc 1.57.0 (f1edd0429 2021-11-29)

info: cleaning up downloads & tmp directories
Removing intermediate container b52352efa9e0
 ---> 375948821afd
Step 11/16 : RUN . $HOME/.cargo/env && pip install -r requirements-docker.txt
 ---> Running in 18cb62482f97
Collecting Flask==1.1.2
  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting Flask-API==2.0
  Downloading Flask_API-2.0-py3-none-any.whl (119 kB)
Collecting Flask-Cors==3.0.10
  Downloading Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB)
Collecting google-cloud==0.34.0
  Downloading google_cloud-0.34.0-py2.py3-none-any.whl (1.8 kB)
Collecting google-cloud-storage==1.35.0
  Downloading google_cloud_storage-1.35.0-py2.py3-none-any.whl (96 kB)
Collecting pandas==1.2.1
  Downloading pandas-1.2.1-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
Collecting torch==1.7.1
  Downloading torch-1.7.1-cp39-cp39-manylinux1_x86_64.whl (776.8 MB)
Collecting transformers==3.3.1
  Downloading transformers-3.3.1-py3-none-any.whl (1.1 MB)
Collecting waitress==1.4.4
  Downloading waitress-1.4.4-py2.py3-none-any.whl (58 kB)
Collecting werkzeug==0.16.0
  Downloading Werkzeug-0.16.0-py2.py3-none-any.whl (327 kB)
Collecting click>=5.1
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
Collecting Jinja2>=2.10.1
  Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting itsdangerous>=0.24
  Downloading itsdangerous-2.0.1-py3-none-any.whl (18 kB)
Collecting Six
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting google-cloud-core<2.0dev,>=1.4.1
  Downloading google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)
Collecting requests<3.0.0dev,>=2.18.0
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting google-auth<2.0dev,>=1.11.0
  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-resumable-media<2.0dev,>=1.2.0
  Downloading google_resumable_media-1.3.3-py2.py3-none-any.whl (75 kB)
Collecting python-dateutil>=2.7.3
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting numpy>=1.16.5
  Downloading numpy-1.22.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Collecting pytz>=2017.3
  Downloading pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting typing-extensions
  Downloading typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Collecting tqdm>=4.27
  Downloading tqdm-4.62.3-py2.py3-none-any.whl (76 kB)
Collecting packaging
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
Collecting filelock
  Downloading filelock-3.4.2-py3-none-any.whl (9.9 kB)
Collecting sentencepiece!=0.1.92
  Downloading sentencepiece-0.1.96-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting tokenizers==0.8.1.rc2
  Downloading tokenizers-0.8.1rc2.tar.gz (97 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting regex!=2019.12.17
  Downloading regex-2021.11.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (763 kB)
Collecting sacremoses
  Downloading sacremoses-0.0.47-py2.py3-none-any.whl (895 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.9/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage==1.35.0->-r requirements-docker.txt (line 5)) (60.5.0)
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.8-py3-none-any.whl (39 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-crc32c<2.0dev,>=1.0
  Downloading google_crc32c-1.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (36 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.10-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Downloading pyparsing-3.0.6-py3-none-any.whl (97 kB)
Collecting joblib
  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
Collecting protobuf>=3.12.0
  Downloading protobuf-3.19.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Downloading googleapis_common_protos-1.54.0-py2.py3-none-any.whl (207 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (pyproject.toml): started
  Building wheel for tokenizers (pyproject.toml): still running...
  Building wheel for tokenizers (pyproject.toml): still running...
  Building wheel for tokenizers (pyproject.toml): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp9cymhzun
       cwd: /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4
  Complete output (332 lines):
  /tmp/pip-build-env-ynxan3h4/overlay/lib/python3.9/site-packages/setuptools/dist.py:493: UserWarning: Normalizing '0.8.1.rc2' to '0.8.1rc2'
    warnings.warn(tmpl.format(**locals()))
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/tokenizers
  copying tokenizers/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers
  creating build/lib.linux-x86_64-3.9/tokenizers/models
  copying tokenizers/models/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers/models
  creating build/lib.linux-x86_64-3.9/tokenizers/decoders
  copying tokenizers/decoders/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers/decoders
  creating build/lib.linux-x86_64-3.9/tokenizers/normalizers
  copying tokenizers/normalizers/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers/normalizers
  creating build/lib.linux-x86_64-3.9/tokenizers/pre_tokenizers
  copying tokenizers/pre_tokenizers/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers/pre_tokenizers
  creating build/lib.linux-x86_64-3.9/tokenizers/processors
  copying tokenizers/processors/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers/processors
  creating build/lib.linux-x86_64-3.9/tokenizers/trainers
  copying tokenizers/trainers/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers/trainers
  creating build/lib.linux-x86_64-3.9/tokenizers/implementations
  copying tokenizers/implementations/byte_level_bpe.py -> build/lib.linux-x86_64-3.9/tokenizers/implementations
  copying tokenizers/implementations/bert_wordpiece.py -> build/lib.linux-x86_64-3.9/tokenizers/implementations
  copying tokenizers/implementations/char_level_bpe.py -> build/lib.linux-x86_64-3.9/tokenizers/implementations
  copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib.linux-x86_64-3.9/tokenizers/implementations
  copying tokenizers/implementations/__init__.py -> build/lib.linux-x86_64-3.9/tokenizers/implementations
  copying tokenizers/implementations/base_tokenizer.py -> build/lib.linux-x86_64-3.9/tokenizers/implementations
  copying tokenizers/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers
  copying tokenizers/models/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers/models
  copying tokenizers/decoders/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers/decoders
  copying tokenizers/normalizers/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers/normalizers
  copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers/pre_tokenizers
  copying tokenizers/processors/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers/processors
  copying tokenizers/trainers/__init__.pyi -> build/lib.linux-x86_64-3.9/tokenizers/trainers
  running build_ext
  info: syncing channel updates for 'nightly-2020-05-14-x86_64-unknown-linux-gnu'
  info: latest update on 2020-05-14, rust version 1.45.0-nightly (75e1463c5 2020-05-13)
  info: downloading component 'cargo'
  info: downloading component 'clippy'
  info: downloading component 'rust-docs'
  info: downloading component 'rust-std'
  info: downloading component 'rustc'
  info: downloading component 'rustfmt'
  info: installing component 'cargo'
  info: installing component 'clippy'
  info: installing component 'rust-docs'
  info: installing component 'rust-std'
  info: installing component 'rustc'
  info: installing component 'rustfmt'
      Updating crates.io index
      Updating git repository `https://github.com/n1t0/rayon-cond`
   Downloading crates ...
    Downloaded rayon-core v1.9.1
    Downloaded rayon v1.5.1
    Downloaded typetag v0.1.8
    Downloaded serde_derive v1.0.133
    Downloaded cfg-if v1.0.0
    Downloaded autocfg v1.0.1
    Downloaded cfg-if v0.1.10
    Downloaded cc v1.0.72
    Downloaded strsim v0.8.0
    Downloaded atty v0.2.14
    Downloaded bitflags v1.3.2
    Downloaded ansi_term v0.12.1
    Downloaded aho-corasick v0.7.18
    Downloaded regex v1.5.4
    Downloaded clap v2.34.0
    Downloaded crossbeam-channel v0.5.2
    Downloaded crossbeam-epoch v0.9.6
    Downloaded ctor v0.1.21
    Downloaded crossbeam-utils v0.8.6
    Downloaded erased-serde v0.3.17
    Downloaded inventory-impl v0.1.11
    Downloaded inventory v0.2.1
    Downloaded indoc-impl v0.3.6
    Downloaded indoc v0.3.6
    Downloaded ghost v0.1.2
    Downloaded getrandom v0.1.16
    Downloaded either v1.6.1
    Downloaded itertools v0.8.2
    Downloaded lock_api v0.3.4
    Downloaded num_cpus v1.13.1
    Downloaded memoffset v0.6.5
    Downloaded memchr v2.4.1
    Downloaded itoa v1.0.1
    Downloaded crossbeam-deque v0.8.1
    Downloaded num-traits v0.2.14
    Downloaded onig v6.3.1
    Downloaded number_prefix v0.3.0
    Downloaded indicatif v0.14.0
    Downloaded ppv-lite86 v0.2.16
    Downloaded pkg-config v0.3.24
    Downloaded parking_lot_core v0.7.2
    Downloaded parking_lot v0.10.2
    Downloaded pyo3cls v0.9.2
    Downloaded pyo3-derive-backend v0.9.2
    Downloaded unindent v0.1.7
    Downloaded typetag-impl v0.1.8
    Downloaded terminal_size v0.1.17
    Downloaded ryu v1.0.9
    Downloaded rand_chacha v0.2.2
    Downloaded unicode_categories v0.1.1
    Downloaded unicode-normalization-alignments v0.1.12
    Downloaded serde_json v1.0.74
    Downloaded pyo3 v0.9.2
    Downloaded paste-impl v0.1.18
    Downloaded unicode-width v0.1.9
    Downloaded smallvec v1.7.0
    Downloaded rand_core v0.5.1
    Downloaded quote v1.0.14
    Downloaded scopeguard v1.1.0
    Downloaded proc-macro-hack v0.5.19
    Downloaded paste v0.1.18
    Downloaded unicode-xid v0.2.2
    Downloaded proc-macro2 v1.0.36
    Downloaded syn v1.0.85
    Downloaded lazy_static v1.4.0
    Downloaded textwrap v0.11.0
    Downloaded serde v1.0.133
    Downloaded inventory v0.1.11
    Downloaded once_cell v1.9.0
    Downloaded rand v0.7.3
    Downloaded console v0.15.0
    Downloaded version_check v0.9.4
    Downloaded vec_map v0.8.2
    Downloaded regex-syntax v0.6.25
    Downloaded libc v0.2.112
    Downloaded onig_sys v69.7.1
   Downloading crates ...
    Downloaded rand_hc v0.2.0
    Downloaded wasi v0.9.0+wasi-snapshot-preview1
    Downloaded cloudabi v0.0.3
    Downloaded hermit-abi v0.1.19
    Downloaded encode_unicode v0.3.6
    Downloaded redox_syscall v0.1.57
    Downloaded winapi v0.3.9
    Downloaded winapi-i686-pc-windows-gnu v0.4.0
    Downloaded winapi-x86_64-pc-windows-gnu v0.4.0
  cargo rustc --lib --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.36
     Compiling unicode-xid v0.2.2
     Compiling syn v1.0.85
     Compiling libc v0.2.112
     Compiling autocfg v1.0.1
     Compiling cfg-if v1.0.0
     Compiling lazy_static v1.4.0
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.36/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=127f0f35c6e17df7 -C extra-filename=-127f0f35c6e17df7 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/proc-macro2-127f0f35c6e17df7 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_xid /root/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=5169222e9a9224f7 -C extra-filename=-5169222e9a9224f7 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.85/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' -C metadata=df69050e300bafff -C extra-filename=-df69050e300bafff --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/syn-df69050e300bafff -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.112/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=6a96d9522760a20a -C extra-filename=-6a96d9522760a20a --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/libc-6a96d9522760a20a -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name autocfg /root/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=4b64b452f871448c -C extra-filename=-4b64b452f871448c --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling memchr v2.4.1
     Compiling serde_derive v1.0.133
     Compiling serde v1.0.133
     Compiling crossbeam-utils v0.8.6
     Compiling scopeguard v1.1.0
       Running `rustc --crate-name cfg_if --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=2deae1b316fdaf6f -C extra-filename=-2deae1b316fdaf6f --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name lazy_static /root/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=79cb8478fcd4b466 -C extra-filename=-79cb8478fcd4b466 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.4.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c28282e528ea8998 -C extra-filename=-c28282e528ea8998 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/memchr-c28282e528ea8998 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /root/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.133/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=7b93b0a1df54dfe9 -C extra-filename=-7b93b0a1df54dfe9 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/serde_derive-7b93b0a1df54dfe9 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /root/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.133/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=f342cb2fe0e2cc0e -C extra-filename=-f342cb2fe0e2cc0e --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/serde-f342cb2fe0e2cc0e -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name scopeguard /root/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=3c7ae9dd20140962 -C extra-filename=-3c7ae9dd20140962 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.8.6/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=d4952fd8dd4f7560 -C extra-filename=-d4952fd8dd4f7560 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/crossbeam-utils-d4952fd8dd4f7560 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling crossbeam-epoch v0.9.6
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.9.6/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=88bc5ce55bc01bb6 -C extra-filename=-88bc5ce55bc01bb6 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/crossbeam-epoch-88bc5ce55bc01bb6 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling proc-macro-hack v0.5.19
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.19/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=235c58df4bfc2d26 -C extra-filename=-235c58df4bfc2d26 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/proc-macro-hack-235c58df4bfc2d26 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling regex-syntax v0.6.25
       Running `rustc --crate-name regex_syntax --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.25/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=79e1cae865a51a95 -C extra-filename=-79e1cae865a51a95 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling rayon-core v1.9.1
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=d2283c246bafe80c -C extra-filename=-d2283c246bafe80c --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/rayon-core-d2283c246bafe80c -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling getrandom v0.1.16
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.16/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=f7f1f89d3879f12a -C extra-filename=-f7f1f89d3879f12a --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/getrandom-f7f1f89d3879f12a -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling serde_json v1.0.74
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.74/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=34c628848631933e -C extra-filename=-34c628848631933e --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/serde_json-34c628848631933e -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling cc v1.0.72
       Running `rustc --crate-name cc --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.72/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=46bea57c09b258af -C extra-filename=-46bea57c09b258af --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling unicode-width v0.1.9
       Running `rustc --crate-name unicode_width /root/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.9/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=bfedf134330d4bf0 -C extra-filename=-bfedf134330d4bf0 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling ryu v1.0.9
       Running `rustc --crate-name ryu --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.9/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=8fa7abc9dff34c50 -C extra-filename=-8fa7abc9dff34c50 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling itoa v1.0.1
       Running `rustc --crate-name itoa --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/itoa-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=34da94ddc3c6a630 -C extra-filename=-34da94ddc3c6a630 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling smallvec v1.7.0
       Running `rustc --crate-name smallvec --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=80382cd50be7e711 -C extra-filename=-80382cd50be7e711 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling pkg-config v0.3.24
       Running `rustc --crate-name pkg_config /root/.cargo/registry/src/github.com-1ecc6299db9ec823/pkg-config-0.3.24/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=92694de5e874cd28 -C extra-filename=-92694de5e874cd28 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling erased-serde v0.3.17
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/erased-serde-0.3.17/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=0af6c29872a5752e -C extra-filename=-0af6c29872a5752e --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/erased-serde-0af6c29872a5752e -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling once_cell v1.9.0
       Running `rustc --crate-name once_cell --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.9.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="race"' --cfg 'feature="std"' -C metadata=a203cc7f4a593743 -C extra-filename=-a203cc7f4a593743 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling either v1.6.1
       Running `rustc --crate-name either /root/.cargo/registry/src/github.com-1ecc6299db9ec823/either-1.6.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=7a8a5a271dc7cb51 -C extra-filename=-7a8a5a271dc7cb51 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling ppv-lite86 v0.2.16
       Running `rustc --crate-name ppv_lite86 --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.16/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=f41c402efdf936bd -C extra-filename=-f41c402efdf936bd --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling unindent v0.1.7
       Running `rustc --crate-name unindent --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/unindent-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=ffdf58ea3b9e347d -C extra-filename=-ffdf58ea3b9e347d --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling version_check v0.9.4
       Running `rustc --crate-name version_check /root/.cargo/registry/src/github.com-1ecc6299db9ec823/version_check-0.9.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=2be5c2cc4c1c8e71 -C extra-filename=-2be5c2cc4c1c8e71 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling cfg-if v0.1.10
       Running `rustc --crate-name cfg_if --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=33adbc7c23a934c7 -C extra-filename=-33adbc7c23a934c7 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling bitflags v1.3.2
       Running `rustc --crate-name bitflags --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.3.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=07e4b438a47db117 -C extra-filename=-07e4b438a47db117 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling inventory v0.1.11
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-0.1.11/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=e086f8ea4f266166 -C extra-filename=-e086f8ea4f266166 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/inventory-e086f8ea4f266166 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling strsim v0.8.0
       Running `rustc --crate-name strsim /root/.cargo/registry/src/github.com-1ecc6299db9ec823/strsim-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=0b9af79a9eabd5cc -C extra-filename=-0b9af79a9eabd5cc --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling number_prefix v0.3.0
       Running `rustc --crate-name number_prefix /root/.cargo/registry/src/github.com-1ecc6299db9ec823/number_prefix-0.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=fa210d0967aff1ff -C extra-filename=-fa210d0967aff1ff --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling vec_map v0.8.2
       Running `rustc --crate-name vec_map /root/.cargo/registry/src/github.com-1ecc6299db9ec823/vec_map-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=99222fb3abba5a69 -C extra-filename=-99222fb3abba5a69 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling ansi_term v0.12.1
       Running `rustc --crate-name ansi_term /root/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.12.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=299089b8a23df6cc -C extra-filename=-299089b8a23df6cc --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling unicode_categories v0.1.1
       Running `rustc --crate-name unicode_categories /root/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode_categories-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=6796d54f4a15ffb4 -C extra-filename=-6796d54f4a15ffb4 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow`
     Compiling lock_api v0.3.4
       Running `rustc --crate-name lock_api --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="nightly"' -C metadata=0e8ba3e8e8af44f0 -C extra-filename=-0e8ba3e8e8af44f0 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern scopeguard=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libscopeguard-3c7ae9dd20140962.rmeta --cap-lints allow`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/crossbeam-utils-d4952fd8dd4f7560/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/rayon-core-d2283c246bafe80c/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/crossbeam-epoch-88bc5ce55bc01bb6/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/memchr-c28282e528ea8998/build-script-build`
     Compiling textwrap v0.11.0
       Running `rustc --crate-name textwrap /root/.cargo/registry/src/github.com-1ecc6299db9ec823/textwrap-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=1e5358ff7f49d80c -C extra-filename=-1e5358ff7f49d80c --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern unicode_width=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --cap-lints allow`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/serde_derive-7b93b0a1df54dfe9/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/serde-f342cb2fe0e2cc0e/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/getrandom-f7f1f89d3879f12a/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/proc-macro-hack-235c58df4bfc2d26/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/syn-df69050e300bafff/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/libc-6a96d9522760a20a/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/proc-macro2-127f0f35c6e17df7/build-script-build`
     Compiling unicode-normalization-alignments v0.1.12
       Running `rustc --crate-name unicode_normalization_alignments /root/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-normalization-alignments-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=b83e8ec0575ef893 -C extra-filename=-b83e8ec0575ef893 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern smallvec=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libsmallvec-80382cd50be7e711.rmeta --cap-lints allow`
     Compiling itertools v0.8.2
       Running `rustc --crate-name itertools /root/.cargo/registry/src/github.com-1ecc6299db9ec823/itertools-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=ea6969d200b46984 -C extra-filename=-ea6969d200b46984 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern either=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libeither-7a8a5a271dc7cb51.rmeta --cap-lints allow`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/serde_json-34c628848631933e/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/erased-serde-0af6c29872a5752e/build-script-build`
       Running `rustc --crate-name crossbeam_utils --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.8.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=45ceb0003524c71c -C extra-filename=-45ceb0003524c71c --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern cfg_if=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern lazy_static=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblazy_static-79cb8478fcd4b466.rmeta --cap-lints allow`
       Running `rustc --crate-name memchr --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.4.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=671cb2e32aa232f7 -C extra-filename=-671cb2e32aa232f7 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/inventory-e086f8ea4f266166/build-script-build`
     Compiling memoffset v0.6.5
       Running `rustc --crate-name build_script_build /root/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.6.5/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=a18cf4779b15d386 -C extra-filename=-a18cf4779b15d386 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/memoffset-a18cf4779b15d386 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern autocfg=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libautocfg-4b64b452f871448c.rlib --cap-lints allow`
     Compiling rayon v1.5.1
       Running `rustc --crate-name build_script_build --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=6090863c6b754e05 -C extra-filename=-6090863c6b754e05 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/rayon-6090863c6b754e05 -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern autocfg=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libautocfg-4b64b452f871448c.rlib --cap-lints allow`
     Compiling num-traits v0.2.14
       Running `rustc --crate-name build_script_build /root/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.14/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=8629b96a01b4610e -C extra-filename=-8629b96a01b4610e --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/num-traits-8629b96a01b4610e -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern autocfg=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libautocfg-4b64b452f871448c.rlib --cap-lints allow`
       Running `rustc --crate-name proc_macro_hack --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.19/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -Cembed-bitcode=no -C metadata=940fbea956d42b7d -C extra-filename=-940fbea956d42b7d --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name libc /root/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.112/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=d1d545bbab16b389 -C extra-filename=-d1d545bbab16b389 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN --cfg libc_cfg_target_vendor --cfg libc_non_exhaustive`
       Running `rustc --crate-name proc_macro2 --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.36/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=4cc4c99b8d0fab22 -C extra-filename=-4cc4c99b8d0fab22 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern unicode_xid=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libunicode_xid-5169222e9a9224f7.rmeta --cap-lints allow --cfg no_literal_from_str --cfg no_group_open_close --cfg no_is_available --cfg use_proc_macro --cfg wrap_proc_macro --cfg proc_macro_span`
     Compiling crossbeam-channel v0.5.2
       Running `rustc --crate-name crossbeam_channel --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-channel-0.5.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="crossbeam-utils"' --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=79a318de3ba8ac08 -C extra-filename=-79a318de3ba8ac08 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern cfg_if=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern crossbeam_utils=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_utils-45ceb0003524c71c.rmeta --cap-lints allow`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/memoffset-a18cf4779b15d386/build-script-build`
     Compiling aho-corasick v0.7.18
       Running `rustc --crate-name aho_corasick --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=157f3d95428c1664 -C extra-filename=-157f3d95428c1664 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern memchr=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libmemchr-671cb2e32aa232f7.rmeta --cap-lints allow`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/rayon-6090863c6b754e05/build-script-build`
       Running `/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/build/num-traits-8629b96a01b4610e/build-script-build`
     Compiling quote v1.0.14
       Running `rustc --crate-name quote --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=ea420e1db2207756 -C extra-filename=-ea420e1db2207756 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern proc_macro2=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libproc_macro2-4cc4c99b8d0fab22.rmeta --cap-lints allow`
       Running `rustc --crate-name memoffset /root/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.6.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=a0f798082f7d4b77 -C extra-filename=-a0f798082f7d4b77 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow --cfg tuple_ty --cfg allow_clippy --cfg maybe_uninit --cfg doctests`
       Running `rustc --crate-name getrandom --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.16/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=22bce8e58e48dc75 -C extra-filename=-22bce8e58e48dc75 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern cfg_if=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern libc=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblibc-d1d545bbab16b389.rmeta --cap-lints allow`
     Compiling num_cpus v1.13.1
       Running `rustc --crate-name num_cpus /root/.cargo/registry/src/github.com-1ecc6299db9ec823/num_cpus-1.13.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=d1eb7316a17ef3b0 -C extra-filename=-d1eb7316a17ef3b0 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern libc=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblibc-d1d545bbab16b389.rmeta --cap-lints allow`
     Compiling terminal_size v0.1.17
       Running `rustc --crate-name terminal_size --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/terminal_size-0.1.17/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=14a22be885faf33f -C extra-filename=-14a22be885faf33f --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern libc=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblibc-d1d545bbab16b389.rmeta --cap-lints allow`
     Compiling parking_lot_core v0.7.2
       Running `rustc --crate-name parking_lot_core --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot_core-0.7.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="nightly"' -C metadata=6839db6955175ef9 -C extra-filename=-6839db6955175ef9 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern cfg_if=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcfg_if-33adbc7c23a934c7.rmeta --extern libc=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblibc-d1d545bbab16b389.rmeta --extern smallvec=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libsmallvec-80382cd50be7e711.rmeta --cap-lints allow`
     Compiling atty v0.2.14
       Running `rustc --crate-name atty /root/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=601625c331c876fb -C extra-filename=-601625c331c876fb --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern libc=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblibc-d1d545bbab16b389.rmeta --cap-lints allow`
       Running `rustc --crate-name crossbeam_epoch --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.9.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=d30c0882f3ed7f6d -C extra-filename=-d30c0882f3ed7f6d --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern cfg_if=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern crossbeam_utils=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_utils-45ceb0003524c71c.rmeta --extern lazy_static=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblazy_static-79cb8478fcd4b466.rmeta --extern memoffset=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libmemoffset-a0f798082f7d4b77.rmeta --extern scopeguard=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libscopeguard-3c7ae9dd20140962.rmeta --cap-lints allow`
       Running `rustc --crate-name syn --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.85/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' -C metadata=b426a749ccd795a4 -C extra-filename=-b426a749ccd795a4 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern proc_macro2=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libproc_macro2-4cc4c99b8d0fab22.rmeta --extern quote=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libquote-ea420e1db2207756.rmeta --extern unicode_xid=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libunicode_xid-5169222e9a9224f7.rmeta --cap-lints allow --cfg syn_no_negative_literal_parse`
       Running `rustc --crate-name num_traits /root/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c8c722b05cb19780 -C extra-filename=-c8c722b05cb19780 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --cap-lints allow --cfg has_i128 --cfg has_to_int_unchecked`
     Compiling rand_core v0.5.1
       Running `rustc --crate-name rand_core --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="getrandom"' --cfg 'feature="std"' -C metadata=41b47b15dbcf2b50 -C extra-filename=-41b47b15dbcf2b50 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern getrandom=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libgetrandom-22bce8e58e48dc75.rmeta --cap-lints allow`
     Compiling clap v2.34.0
       Running `rustc --crate-name clap --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=d1d931e823cd42c4 -C extra-filename=-d1d931e823cd42c4 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern ansi_term=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libansi_term-299089b8a23df6cc.rmeta --extern atty=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libatty-601625c331c876fb.rmeta --extern bitflags=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libbitflags-07e4b438a47db117.rmeta --extern strsim=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libstrsim-0b9af79a9eabd5cc.rmeta --extern textwrap=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libtextwrap-1e5358ff7f49d80c.rmeta --extern unicode_width=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --extern vec_map=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libvec_map-99222fb3abba5a69.rmeta --cap-lints allow`
     Compiling parking_lot v0.10.2
       Running `rustc --crate-name parking_lot --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.10.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="nightly"' -C metadata=c06e3469bd781de6 -C extra-filename=-c06e3469bd781de6 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern lock_api=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblock_api-0e8ba3e8e8af44f0.rmeta --extern parking_lot_core=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libparking_lot_core-6839db6955175ef9.rmeta --cap-lints allow`
  error[E0658]: `if` is not allowed in a `const fn`
    --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/app/settings.rs:7:1
     |
  7  | / bitflags! {
  8  | |     struct Flags: u64 {
  9  | |         const SC_NEGATE_REQS       = 1;
  10 | |         const SC_REQUIRED          = 1 << 1;
  ...  |
  51 | |     }
  52 | | }
     | |_^
     |
     = note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
     = help: add `#![feature(const_if_match)]` to the crate attributes to enable
     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

  error[E0658]: `if` is not allowed in a `const fn`
    --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/args/settings.rs:6:1
     |
  6  | / bitflags! {
  7  | |     struct Flags: u32 {
  8  | |         const REQUIRED         = 1;
  9  | |         const MULTIPLE         = 1 << 1;
  ...  |
  28 | |     }
  29 | | }
     | |_^
     |
     = note: see issue #49146 <https://github.com/rust-lang/rust/issues/49146> for more information
     = help: add `#![feature(const_if_match)]` to the crate attributes to enable
     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

     Compiling regex v1.5.4
       Running `rustc --crate-name regex --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.5.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=1e333a3fef530f25 -C extra-filename=-1e333a3fef530f25 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern aho_corasick=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libaho_corasick-157f3d95428c1664.rmeta --extern memchr=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libmemchr-671cb2e32aa232f7.rmeta --extern regex_syntax=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libregex_syntax-79e1cae865a51a95.rmeta --cap-lints allow`
     Compiling crossbeam-deque v0.8.1
       Running `rustc --crate-name crossbeam_deque --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="crossbeam-epoch"' --cfg 'feature="crossbeam-utils"' --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=98dbc32454e4e8b2 -C extra-filename=-98dbc32454e4e8b2 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern cfg_if=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcfg_if-2deae1b316fdaf6f.rmeta --extern crossbeam_epoch=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_epoch-d30c0882f3ed7f6d.rmeta --extern crossbeam_utils=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_utils-45ceb0003524c71c.rmeta --cap-lints allow`
     Compiling rand_chacha v0.2.2
       Running `rustc --crate-name rand_chacha --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_chacha-0.2.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=e7ad19857ddbc1d9 -C extra-filename=-e7ad19857ddbc1d9 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern ppv_lite86=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libppv_lite86-f41c402efdf936bd.rmeta --extern rand_core=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/librand_core-41b47b15dbcf2b50.rmeta --cap-lints allow`
       Running `rustc --crate-name rayon_core --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=af1f8b9b5f2ff68e -C extra-filename=-af1f8b9b5f2ff68e --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern crossbeam_channel=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_channel-79a318de3ba8ac08.rmeta --extern crossbeam_deque=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_deque-98dbc32454e4e8b2.rmeta --extern crossbeam_utils=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libcrossbeam_utils-45ceb0003524c71c.rmeta --extern lazy_static=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblazy_static-79cb8478fcd4b466.rmeta --extern num_cpus=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libnum_cpus-d1eb7316a17ef3b0.rmeta --cap-lints allow`
     Compiling paste-impl v0.1.18
       Running `rustc --crate-name paste_impl --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -Cembed-bitcode=no -C metadata=7047fcfa649dd5ba -C extra-filename=-7047fcfa649dd5ba --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern proc_macro_hack=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libproc_macro_hack-940fbea956d42b7d.so --extern proc_macro --cap-lints allow`
  error: aborting due to 2 previous errors

  For more information about this error, try `rustc --explain E0658`.
     Compiling rand v0.7.3
  error: could not compile `clap`.

  Caused by:
    process didn't exit successfully: `rustc --crate-name clap --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.34.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=d1d931e823cd42c4 -C extra-filename=-d1d931e823cd42c4 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern ansi_term=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libansi_term-299089b8a23df6cc.rmeta --extern atty=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libatty-601625c331c876fb.rmeta --extern bitflags=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libbitflags-07e4b438a47db117.rmeta --extern strsim=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libstrsim-0b9af79a9eabd5cc.rmeta --extern textwrap=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libtextwrap-1e5358ff7f49d80c.rmeta --extern unicode_width=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libunicode_width-bfedf134330d4bf0.rmeta --extern vec_map=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libvec_map-99222fb3abba5a69.rmeta --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
       Running `rustc --crate-name rand --edition=2018 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="getrandom"' --cfg 'feature="getrandom_package"' --cfg 'feature="libc"' --cfg 'feature="std"' -C metadata=b6e3f777bcce4393 -C extra-filename=-b6e3f777bcce4393 --out-dir /tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps -L dependency=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps --extern getrandom_package=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/libgetrandom-22bce8e58e48dc75.rmeta --extern libc=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/liblibc-d1d545bbab16b389.rmeta --extern rand_chacha=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/librand_chacha-e7ad19857ddbc1d9.rmeta --extern rand_core=/tmp/pip-install-y729o32a/tokenizers_c53f7559dfd942c688aa3a08e36d7bb4/target/release/deps/librand_core-41b47b15dbcf2b50.rmeta --cap-lints allow`
  error: build failed
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
The command '/bin/sh -c . $HOME/.cargo/env && pip install -r requirements-docker.txt' returned a non-zero code: 1
kaaquist commented 2 years ago

I found a workaround that works for me. I bumped down my python version to version 3.8.12. Since there is pre-build binaries done for that version on linux, I could pip install ... and everything was good for me. I do think that this here error has something to do with rustc that has changed something. How to fix it I'm not sure, but please let me know if I need to provide more information on my setup. Cheers.

Narsil commented 2 years ago

Thanks for all this information.

Glad you could find a workaround that issue I am pretty sure it could help other users.

github-actions[bot] commented 8 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.