WordPieceTokenizer in Chinese can not use.

nlpodyssey / cybertron

Cybertron: the home planet of the Transformers in Go

BSD 2-Clause "Simplified" License

289 stars 26 forks source link

Open sunhailin-Leo opened 2 years ago

sunhailin-Leo commented 2 years ago

matteo-grella commented 2 years ago

Thanks @sunhailin-Leo we understood the issue and we are going to fix it asap