rigatoni / linguine

Issue tracking repository.
http://rigatoni.se.rit.edu
3 stars 2 forks source link

Cleanup operations are not receiving tokenized word content before they run #98

Open danrlavoie opened 9 years ago

danrlavoie commented 9 years ago

https://github.com/rigatoni/linguine-python/blob/dev/linguine/transaction.py Looking mainly in lines 55-75 of this file, Cleanups are not checked for if they are in the list of token-based operations, and they always work on the corpus rather than the tokenized corpus. The code needs to be changed to fix that issue.