Tokenization

[mantono]

What should be done? Transform all unstructured text data to a set of strings.

When is it finished?

• [x] All characters separated by non-letter character are broken down to words
  • [x] Whitespace
  • [x] Punctunations
  • [x] ":" and ";"
• [x] All input is transformed to lower case

Issues it depends on: #3