Closed rth closed 5 years ago
Add a preliminary implementation of VTextTokenizer, that implements manual rules on top of unicode-segmentation aiming to improve tokenization accuracy on the Universal Dependencies corpus tokenization.
Add a preliminary implementation of VTextTokenizer, that implements manual rules on top of unicode-segmentation aiming to improve tokenization accuracy on the Universal Dependencies corpus tokenization.