end_of_word_suffix = "</w>" no work??

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

https://huggingface.co/docs/tokenizers

Apache License 2.0

8.69k stars 747 forks source link

end_of_word_suffix = "</w>" no work?? #1372

Closed longday1102 closed 7 months ago

longday1102 commented 8 months ago

Why did I use the parameter end_of_word_suffix = "</w>" but when training is finished, I check the tokenizer.json file and still see end_of_word_suffix = null and the words in the vocab do not have that suffix added??

ArthurZucker commented 8 months ago

Hey, could you share a reproducer? 🤗

longday1102 commented 8 months ago

Hey, could you share a reproducer? 🤗

oh I'm sorry, I've solved this problem, thank you for your reply 😄😄

github-actions[bot] commented 7 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.