mideind / Tokenizer

A tokenizer for Icelandic text
Other
27 stars 6 forks source link

Support for citation characters #16

Open sveinbjornt opened 4 years ago

sveinbjornt commented 4 years ago

The tokenizer should support superscripted citation characters. This will also help with GreynirCorrect, which I assume will be heavily used to read student essays and academic papers.

Screen Shot 2020-06-30 at 23 14 20