Closed rth closed 5 years ago
This adds some specific rules for french tokenization, and improves the language independent tokenizer following #28 .
Latest UD treebank evaluation results are,
tokenizer regexp spacy unicode-segmentation vtext lang treebank de GSD 0.818 0.934 0.953 0.947 en EWT 0.743 0.975 0.927 0.968 GUM 0.807 0.994 0.980 0.997 fr Sequoia 0.748 0.946 0.861 0.943
This adds some specific rules for french tokenization, and improves the language independent tokenizer following #28 .
Latest UD treebank evaluation results are,