tokenizer.ngrams_from_doc does not use stopwords

scottkleinman / lexos

Development repo for the Lexos API

MIT License

1 stars 0 forks source link

tokenizer.ngrams_from_doc does not use stopwords #19

Closed JeffreyYStewart closed 1 year ago

JeffreyYStewart commented 2 years ago

tokenizer.ngrams_from_doc will not include tokens marked as stopwords in the ngrams it produces.

scottkleinman commented 1 year ago

tokenizer.ngrams_from_doc now accepts any argument that can be passed to textacy.extract.basics.ngrams. This allows the user to choose whether stop words (or other features) will be included.