search

mideind / Tokenizer

A tokenizer for Icelandic text
Other
27 stars 6 forks source link

Two dots #36

Closed starkadur closed 2 years ago

starkadur commented 3 years ago

Sentence that ends with two dots is split into two sentences while a sentence that ends with three dots is not: "This is a sentence.." becomes "This is a sentence. ¦ ." while "This is a sentence..." becomes "This is a sentence ..."

Holado commented 2 years ago

This has been fixed in the newest version.