check token type for EOF before converting

ahmetaa / zemberek-nlp

NLP tools for Turkish.

Other

1.14k stars 208 forks source link

check token type for EOF before converting #239

Closed otuncelli closed 4 years ago

otuncelli commented 4 years ago

hasNext() throws if the EOF token comes after an ignored token.

Caused by: java.lang.IllegalStateException: Unidentified token type =EOF at zemberek.tokenization.TurkishTokenizer.convertType(TurkishTokenizer.java:161) ~[?:?] at zemberek.tokenization.TurkishTokenizer$TokenIterator.hasNext(TurkishTokenizer.java:219) ~[?:?]

ahmetaa commented 4 years ago

Thanks