Disallow near-identical words

[jonadsimon]

Don't want multiple near-identical words appearing in the word search

Already take one step in this direction by disallowing words that are superstrings of other words

However this does not capture cases of subtly different spelling e.g. color/colour, ebonise/ebonize, etc

Should add a Levenshtein-distance metric enforcing edit distance be > 1

[jonadsimon]

Not enough to limit to Levenshtein-distance > 1, need to also exclude words with share too-high proportion of their letters (or something equivalent to this)

Nearest neighbors for oligodendrocyte include:

1. oligodendrocytes : excluded by substring-matching only if we include seed word
2. oligodendrocytic : not excluded
3. Oligodendrocyte : excluded by downcasing equivalence with oligodendrocyte
4. oligodendroglial : excluded as superstring with oligodendroglia
5. oligodendrogenesis : not excluded
6. oligodendroglia : not excluded

[jonadsimon]

Fixed for simple cases via addition of Levenstein constraint + 70% matching constraint. However still wouldn't for the long suffixes on "oligodendrocytes". Marking as fixed until otherwise noted

[jonadsimon]

Still has issues with near-identical short words: "marked" vs "marking", "time" vs "timing"

Use an out-of-the-box stemmer to identify base words and remove duplicates that way

[jonadsimon]

Add word-length constrain to Levenstein distance to only apply to words of length ≥5. Currently it's misfiring on e.g. "time" vs "tide"

[jonadsimon]

Added Levenstein min-length and stemming