Closed paul999 closed 7 years ago
We need to be carefull when merging this, it changes the full tokenizer, and we have very limited (Now a bit more) test coverage there. We should wait a while with actually making a release with these changes to make sure everything still works.
We need to be carefull when merging this, it changes the full tokenizer, and we have very limited (Now a bit more) test coverage there. We should wait a while with actually making a release with these changes to make sure everything still works.