Closed kilimchoi closed 4 months ago
Is it possible to update the tokenizers' version? We're using the latest transformers python package and it requires tokenizers' version to be >=0.19,<0.20
Hey, bumped it to ^0.19 in 5.3.3, thanks for reporting!
Hi, This issue got resolved with 5.3.3 but again pops up with the latest 5.5.6
Is it possible to update the tokenizers' version? We're using the latest transformers python package and it requires tokenizers' version to be >=0.19,<0.20