search

TimSchopf / KeyphraseVectorizers

Set of vectorizers that extract keyphrases with part-of-speech patterns from a collection of text documents and convert them into a document-keyphrase matrix.
https://arxiv.org/abs/2210.05245
BSD 3-Clause "New" or "Revised" License
254 stars 34 forks source link

Expose regex token_pattern #20

Open raj-shah opened 1 year ago

raj-shah commented 1 year ago

Hello!

Curious if it would be possible to expose a regex token pattern param like that in CountVectorizer? This would help in filtering for (un)wanted chars during tokenization, e.g. hyphens, ampersands, apostrophes, etc.

The workaround I have found so far has been to use a custom POS tagger (custom_pos_tagger param of KeyphraseVectorizer) wherein I don't change any POS patterns/behaviour but recompile and modify the underlying spacy tokenizer object's prefix, suffix, and infix params. Wondering if there is a simpler way of exposing such behaviour? Keen to hear your thoughts!

Thanks

andreivintila10 commented 1 year ago

I'm running into the same issue here where I need to consider hyphenated compound words within key-phrases. Using the English model in spaCy I've managed to remove the infix hyphen splitting rule from the tokenizer before passing the model to the KeyphraseVectorizer. Then, further tracked it down to where it performs the transform on the CountVectorizer and the compound words are being discarded because they are not matching the default token_pattern.

I've worked on this here: https://github.com/andreivintila10/KeyphraseVectorizers