Improvements to dictionary matching

frreiss commented 3 years ago

This PR includes some fixes for issues with our dictionary matching that encountered while working on a market intelligence use case for a blog post.

The specific problems addressed here are:

There was no API for creating a dictionary from an in-memory list, so I've added a new function create_dict().
SpaCy's default set tokenizers tend to overthink how they treat punctuation, which leads to dictionary entries getting tokenized differently from text, which messes up dictionary matching. So I've added a function simple_tokenizer() that returns a tokenizer that splits on every chunk of whitespace and on every punctuation character. Dictionary creation uses that tokenizer by default now.

I also fixed a minor bug in the handling of the warnings element of responses from Watson Natural Language Understanding.

frreiss commented 3 years ago

@BryanCutler would you mind giving these changes a quick review?

Thanks!

BryanCutler commented 3 years ago

Sorry, slipped under the radar. Looking at it now.

frreiss commented 3 years ago

Thanks for the review! Pushed some corrections. Will merge once this branch passes tests.

CODAIT / text-extensions-for-pandas