huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
https://huggingface.co/docs/tokenizers
Apache License 2.0
8.68k stars 745 forks source link

Failing to build bindings with 0.19.1 #1505

Closed bryteise closed 4 days ago

bryteise commented 2 months ago

With the 0.19.1 release I get the following error:

-> % python -m build --wheel                                                                                                                                                                                                                  
* Creating isolated environment: venv+pip...                                                                                                                                                                                                  
* Installing packages in isolated environment:                                                                                                                                                                                                
  - maturin>=1.0,<2.0                                                                                                                                                                                                                         
* Getting build dependencies for wheel...                                                                                                                                                                                                     
* Building wheel...                                                                                                                                                                                                                           
Running `maturin pep517 build-wheel -i /tmp/build-env-2cqz6gzk/bin/python --compatibility off`                                                                                                                                                
error: failed to select a version for `env_logger`.                                                                                                                                                                                           
    ... required by package `tokenizers-python v0.19.1 (/home/wdouglas/clearlinux/packages/pypi-tokenizers/tokenizers-0.19.1/bindings/python)`                                                                                                
versions that meet the requirements `^0.11` (locked to 0.11.3) are: 0.11.3                                                                                                                                                                    

the package `tokenizers-python` depends on `env_logger`, with features: `anstream` but `env_logger` does not have these features.                                                                                                             
 It has an optional dependency with that name, but that dependency uses the "dep:" syntax in the features table, so it does not have an implicit feature with that name.                                                                      

failed to select a version for `env_logger` which could resolve this conflict                                                                                                                                                                 
💥 maturin failed                                                                                                                                                                                                                             
  Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?                                                                                                                                                               
  Caused by: `cargo metadata` exited with an error:                                                                    
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/build-env-2cqz6gzk/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

ERROR Backend subprocess exited when trying to invoke build_wheel

And it looks like the issue is the Cargo.toml enforcing the use of env_logger 0.11 as there is some incompatability with anstream. If I update the config to use any version of env_logger it select 0.10 and the bindings are able to build.

ArthurZucker commented 2 months ago

It’s weird since we used this for the recent release. Will check

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

bryteise commented 1 month ago

Did you see anything? I am seeing the same error though I did update maturin.

bryteise commented 1 month ago

Oh I think I understand the issue. I was using a single cargo vendor but this has two rust repos in it which don't share the same dependencies.

In order to build offline I try to keep a single vendor per module and this breaks that assumption.

github-actions[bot] commented 1 week ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.