Find/Create function for text tokenzation

[AndreyKondakovGW]

Find/Create function for sepatating text string to tokens(words). Function must get text string and return list of string tokens. Function also should not return tokens containing digits and punctuation (you can do this with regular expressions). Function must have optional parameters:

1. to_lower - (shows that all tokens in returnable list must be in lower case)
2. min_token_size - (shows min length of returnable tokens)

[AndreyKondakovGW]

Exaple of this function: (on their basis you can write test fo this function)

1. tokenize(text="Классификация текстов (документов) (англ. Document classification) — задача компьютерной лингвистики") => ["Классификация", "текстов", "документов" , "англ", "Document", classification", "задача", "компьютерной", "лингвистики"]
2. tokenize(text="Классификация настроения текста из базы ANEW[3], : счастливый - 8.21; хороший - 7.47; скучный - 2.95;", to_lower = true, min_token_size=4) => ["классификация", "настроения", "текста", "базы", "anew", "счастливый", "хороший", "скучный"]

[CyberSniff]

I take it

[AndreyKondakovGW]

Also sugest add lemmatization option to lemmatize tokens, as independant function or as part of tokenize function tokenize(text="Классификация настроения текста из базы , to_lower = true, min_token_size=4, lemmatize = True) => ["классификация", "настроение", "текст", "база"]

[Wolwer1nE]

Looks like there is no ready to use library for lemmatization written in ruby, so we will focus on tokenzation in this issue and extract lemmatization as a separate issue.