search

bhavnicksm / chonkie

🦛 CHONK your texts with Chonkie ✨ - The no-nonsense RAG chunking library
https://pypi.org/project/chonkie/
MIT License
1.68k stars 60 forks source link

ImportError: cannot import name 'tokenizer' from 'tokenizers' (/usr/local/lib/python3.10/site-packages/tokenizers/__init__.py) #25

Closed abchbx closed 2 weeks ago

abchbx commented 2 weeks ago

怎么解决？

bhavnicksm commented 2 weeks ago

Hey @abchbx!

You might be making a minor spelling mistake here, could you try with Tokenizer (capital T) instead, like this:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("gpt2")

Hope this resolves the issue, closing the issue but you may re-open if you're still facing the same issue.