Issue with Latin Memorizing tokenizer

hipster-philology / nlp-pie-taggers

Extension for pie to include taggers with their models and pre/postprocessors

Mozilla Public License 2.0

11 stars 2 forks source link

Issue with Latin Memorizing tokenizer #1

Closed PonteIneptique closed 4 years ago

PonteIneptique commented 4 years ago

Exception: The output token does not match our inputs _ : laetatur

Need to investigate this bug, but to do that, I also need to make sure that when one file throws an error, we know which one. Because right now, it's one of the 15k files I lemmatized...

PonteIneptique commented 4 years ago

Apparently, two differrent issues:

one with underscore breaking the memorizingtokenizer
one with "max() arg is an empty sequence"