In the Tokenizer package, there are parameters which are not used in base class but rather in the specialized classes.
To make the API consistent, use those parameters in base methods, for example, use_stopwords can be used to trigger filtering in base tokenizer method. Similarly, for min_token_length.
In the Tokenizer package, there are parameters which are not used in base class but rather in the specialized classes. To make the API consistent, use those parameters in base methods, for example, use_stopwords can be used to trigger filtering in base tokenizer method. Similarly, for min_token_length.