search

google / sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.
Apache License 2.0
10.2k stars 1.17k forks source link

Compatibility Issue when using v0.2.0 with transformers and tensorflow #1060

Open aws-tianquaw opened 2 weeks ago

aws-tianquaw commented 2 weeks ago

I recently encountered some compatibility issue when using sentencepiece v0.2.0 together with latest transformers and tensorflow packages. When I ran some Python script that imports AutoProcessor class from transformers, the Python kernel was killed with the following error message:

ERROR: Flag 'minloglevel' was defined more than once (in files 'src/error.cc' and 'home/conda/feedstock_root/build_artifacts/abseil-split_1720857154496/work/absl/log/flags.cc').

I tried to downgrade sentencepiece to v0.1.99 and then the issue was solved.

Commands to reproduce the issue:

conda create -n "py311" python=3.11
source activate py311
conda install -c conda-forge transformers==4.45 tensorflow==2.17 sentencepiece==0.2
python -c "from transformers import AutoProcessor"
Genjidegaulle commented 2 days ago

Running into a similar issue, with the same temporary resolution of downgrading sentencepiece to 0.1.99.