WordPieceTokenizer in Chinese can not use.

nlpodyssey / cybertron

Cybertron: the home planet of the Transformers in Go

BSD 2-Clause "Simplified" License

280 stars 26 forks source link

Open sunhailin-Leo opened 1 year ago

sunhailin-Leo commented 1 year ago

matteo-grella commented 1 year ago

Thanks @sunhailin-Leo we understood the issue and we are going to fix it asap