Create stem dictionary from training and test set

[freezing]

Stem dictionary can be created using only words from the training and test data.

Some dummy algorithm could look like:

• get all the words from train.csv and test.csv and sort them alphabetically
• words that have the same stem should be next to each other, so find intervals by checking the length of common prefix and maybe Levenshtein distance

[freezing]

https://xapian.org/docs/stemming.html

[freezing]

This can help with plurals.

[freezing]

Very interesting for word similarity: http://orion.lcg.ufrj.br/Dr.Dobbs/books/book5/chap08.htm