mideind / Tokenizer

A tokenizer for Icelandic text
Other
27 stars 6 forks source link

Add a META_BEGIN token kind #42

Closed peturorri closed 2 years ago

peturorri commented 2 years ago

This helps users distinguish between metatokens and regular tokens

peturorri commented 2 years ago

I chose 9999 to make < and <= do the same thing when filtering. Less surprising and less prone to bugs that way (in my opinion ;).