search

vseloved / cl-nlp

Common Lisp NLP toolset
Other
219 stars 28 forks source link

Condition handler for multiple sentences for word tokenizer? #7

Closed dmsurti closed 9 years ago

dmsurti commented 9 years ago

As per the current word tokenizer contract, it handles only single sentence. Should we add a condition handler if multiple sentences are submitted for tokenization?

vseloved commented 9 years ago

I think, it's impossible to know that for sure.

dmsurti commented 9 years ago

Ahh, looking at this example:

I called Dr. Jones. I called Dr. Jones.[>>>]I called Dr. Jones . I called Dr. Jones .

and looking carefully at Dr. / Jones. , I think I know why it is impossible for sure!!!.

Closing the issue.