Tokenizer testing - Githubissues

elitcloud / elit

🔠 Evolution of Language and Information Technology

https://elit.cloud

Other

47 stars 7 forks source link

Tokenizer testing #12

Closed cathxiao closed 7 years ago

cathxiao commented 7 years ago

Proposed Changes

addition of code to handle colloquial contractions such as "'em", "'tis", etc. (i.e. words with apostrophes inside)
addition of unit tests for the tokenizer.

imgarylai commented 7 years ago

@cathxiao Thanks for this PR. I'm doing rebase and I have a question. Is elit/cpp/catch.hpp an automatically generated file for TEST? If it is, I will put this into .gitignore.

cathxiao commented 7 years ago

@imgarylai No, elit/cpp/catch.hpp is a unit testing library, and elit/cpp/main.cpp depends on it.