hasNext() throws if the EOF token comes after an ignored token.
Caused by: java.lang.IllegalStateException: Unidentified token type =EOF at zemberek.tokenization.TurkishTokenizer.convertType(TurkishTokenizer.java:161) ~[?:?] at zemberek.tokenization.TurkishTokenizer$TokenIterator.hasNext(TurkishTokenizer.java:219) ~[?:?]
hasNext() throws if the EOF token comes after an ignored token.
Caused by: java.lang.IllegalStateException: Unidentified token type =EOF at zemberek.tokenization.TurkishTokenizer.convertType(TurkishTokenizer.java:161) ~[?:?] at zemberek.tokenization.TurkishTokenizer$TokenIterator.hasNext(TurkishTokenizer.java:219) ~[?:?]