Tokenization parameter - Githubissues

tomerm / MLClassification

Classification using ML approach for English / Hebrew / Arabic data sets

1 stars 2 forks source link

Tokenization parameter #30

Open matanzuckerman opened 5 years ago

matanzuckerman commented 5 years ago

Hi @tomerm @semion1956 As discussed before, I would like to have another boolean parameter in the Tokenization process. The parameter will define if we want to do Tokenization or not. Now I have to run Tokenization in order to remove stop words. There are some cases I would like to remove stop words and not remove POS.

Thanks

tomerm commented 5 years ago

It was already introduced:

actualTocs = yes/no

If you wish to have stop words removed but without tokenization please use:

actualTocs = no; stopWords = yes