Closed milmor closed 3 weeks ago
Sorry guys, I've found the bug, is due '\xa0'. How can I fix this? In the text file is saved as a blank space.
I have fixed this haha, just use tf_text.normalize_utf8(text, 'NFKD') to standardize the text before creating your vocab. Sorry guys.
Thanks @milmor this is actually quite interesting!
@mattdangerw should we include a note in the docstring? It seems difficult to catch the exception without a lot of overhead.
Yeah this is interesting, thanks! @milmor is it possible to repro the issue in a colab? So we could take a look at the fix you applied and issue without the fix?
When passing 'test_wiki' filename to WordPieceTokenize:
get the following error:
but when passing exactly the same vocabulary in list format, there is no error:
I'm saving the vocab created by:
with the following: