Add word cooccurrence as intent classification features

Goal

During experiments done to improve intent classification, using word cooccurrences as additional features to tf-idf was found to be helpful to improve classification performances in cases where words transitions and/or order is meaningful.

For intance, a model using a bag-of-unigram tf-idf features struggle to differentiate the following utterances: "turn on the light" (TurnLightsOn) and "is the light turned on" (CheckLightsStatus).

Adding order cooccurence feature can help in such cases.

Work done

1. Add optional cooccurence features to the intent classification

Transformed the Featurizer object into a ProcessingUnit. The featurizer now relies on a TfidfVectorizer ProcessingUnit (wrapping the sklearn TfidfVectorizer) and optionally on a CooccurrenceVectorizer.

The feature extraction works as following:

tf-idf features are extracter from the dataset
feature selection is applied to select the best feature
if configuration added_cooccurrence_feature_ratio parameter is > 0 the cooccurence features will be added. The CooccurrenceVectorizer is fitted on all non null utterances and then cooccurrence features are ranked using the same feature selection as for tf-idf features. The top-k cooccurence features will be used. k is computed using the added_cooccurrence_feature_ratio config parameter: if we have n tf-idf features then at most int(n * added_cooccurrence_feature_ratio) will be added in the feature matrix

Note: the order of the words is kept when computing word cooccurrence. Also note that the user can restrict the size of the window in which we'll look for cooccurrence

2. Miscellaneoous

Refactor the EntityParser class to make it more generic, and to ease implementation of other entity parsers

Important note: this PR is breaking

Checklist:

[x] My PR is ready for code review
[x] I have added some tests, if applicable, and run the whole test suite, including linting tests
[x] I have updated the documentation, if applicable

snipsco / snips-nlu