Spell Checker

[freezing]

• Correct spell errors in the tokens.
  • These should fix the following:
  • - batf -> bath; wrong letter
  • - sponge -> spounge; missing letter
  • - lightbulb -> light bulb; space error - in this case it is split in two tokens (order MUST be preserved), or this can be done in next level cleaner (idea is similar to spell checker's)
  • - 1x3 -> 1 x 3

Create a job that will create spell checker dictionary, each entry is:

• word with removed letters -> list of original words
• e.g. (ba. bt, bh, at, ah, th, bat, bah, ath, bath) -> bath

Maybe remove only 1 letter for words shorter than 5, and don't spell correct words with length <= 2?

[freezing]

Spell checker should use words from the product titles from both train and test data, as well as attributes and descriptions - we are assuming that these are well written.

[freezing]

http://blog.faroo.com/2012/06/07/improved-edit-distance-based-spelling-correction/

[freezing]

https://www.quora.com/What-are-some-algorithms-of-spelling-correction-that-were-used-by-search-engine

[freezing]

http://norvig.com/spell-correct.html

[freezing]

def smallErrors(w: String): List[String] = {
    smallErrors(w, 1) union smallErrors(w, 2)
  }
  // TODO: Add memoization
  def smallErrors(w: String, d: Int): List[String] = {
    if (d > w.length) throw new IllegalArgumentException("Nu")
    d match {
      case 0 => List(w)
      case 1 => smallErrors1(w)
      case k => smallErrors1(w) flatMap { s => smallErrors(s, d - 1) }
    }
  }
  def smallErrors1(w: String): List[String] = {
    (0 until w.length map { idx => w.substring(0, idx) + w.substring(idx + 1) }).toList
  }