search

mideind / Tokenizer

A tokenizer for Icelandic text
Other
27 stars 6 forks source link

Token stream wrapper #35

Closed sultur closed 3 years ago

sultur commented 3 years ago

Idea for a TokenStream wrapper class which contains an iterator and a small deque used for lookahead.