Open dongteng opened 3 years ago
Please update transformers and the tokenizer python packages.
I've solved it by updating the tokenizers to the latest
Please update transformers and the tokenizer python packages.
Thanks!!
Hi, just to add that this seems to be a missing dependency in sentence-transformers
. Perhaps you want to add tokenizers>=0.10.3
explicitly if that is the minimal version you need for (all) your models to work. You only have the transformers
dependency at the moment, but transformers itself is less strict (tokenizers>=0.10.1,<0.11
).
@buhrmann Thanks for pointing this out, updated the requirements
Can someone advise me how to I update the tokenizer to the latest? I am using miniforge on an m1 mac and I cannot make progress pass this error. I have tried pip install tokenizers=0.10.3
after doing conda install sentence-transformers
.
Many thanks in advance.
try pip install -U tokenizers
It is also präferable to install sentence-transformers via pip, not with conda: Conda people are sadly unreliable to update the versions. Pull requests are often ignored for quite a while.
thanks - unfortunately, that does not work. If I try to install using sentence-transformers using the pip route, it fails hard - because of the m1 silicon. scikit-learn will just not build. I have decided not to ML on my mac for now and use my other windows / intel HW.
need help.thanks!!! model = SentenceTransformer('/dataset/BERT/paraphrase-multilingual-mpnet-base-v2') I've downloaded the pretrained model because I want to use it offline.Then the error