TokenRole has no attribute ENDOFQUOTE

proycon / python-ucto

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).

29 stars 5 forks source link

TokenRole has no attribute ENDOFQUOTE #4

Closed emanjavacas closed 6 years ago

emanjavacas commented 6 years ago

Hi there, I noticed that isendofquote seems to be broken.

Seems like a typo on this line:

https://github.com/proycon/python-ucto/blob/65a7f03a92f60fa28e330a5fb735d75230cdbec4/ucto_wrapper.pyx#L29

which should be rather ENDOFQUOTE.

proycon commented 6 years ago

Right! Well spotted, will fix. Thanks! (quote detection is fortunately disabled by default though)