search

mideind / Tokenizer

A tokenizer for Icelandic text
Other
27 stars 6 forks source link

Number tokenization #30

Closed sultur closed 3 years ago

sultur commented 3 years ago

Modifications to tokenizer which makes parsing written numbers less aggressive. (Also typo fixes and whitespace fixes in Abbrev.conf)