Whitespace can confuse StringMatchingRecommender

[reckart]

Describe the bug If the StringMatchingRecommender learns that John Smith is a person, it won't predict that John\tSmith is a person as well.

To Reproduce Steps to reproduce the behavior:

1. Import John Smith John\tSmith and test.

Expected behavior The kind of whitespace which separates tokens should not confuse the recommender.

Please complete the following information:

• Version and build ID: 4a9f918d75a0ac64af72dbc61b0398c61860f8e2

[jcklie]

Dont we normalize whitespace somewhere?

[reckart]

Not at the level of the document text that is in the CAS. The CAS contains whatever whitespace the original document contained (and which passed through the DKPro Core Reader). We do a bit of normalization e.g. when sending text to brat.

[reckart]

Note, for other recommenders, this shouldn't be a problem because they operate on Tokens and Tokens normally don't include whitespace. But the StringMatchingRecommender operates directly on the CAS document text.

[jcklie]

What happens if we would introduce the sentence level recommender in #590 ? Do we get the bad whitespace from the sentence?

[reckart]

Well - normally a recommender would operate on tokens, not on the base text. But for the StringMatchingRecommender, it is actually easier applying the Trie directly to the base text instead of first constructing a string from the tokens.

[reckart]

@Rentier for a sentence-level recommender cf. OpenNlpDoccatRecommender.