search

TimSchopf / KeyphraseVectorizers

Set of vectorizers that extract keyphrases with part-of-speech patterns from a collection of text documents and convert them into a document-keyphrase matrix.
https://arxiv.org/abs/2210.05245
BSD 3-Clause "New" or "Revised" License
254 stars 35 forks source link

Lemmatizing documents and keyphrases #9

Open hboisgibault opened 2 years ago

hboisgibault commented 2 years ago

Using lemmatization can result in better quality keyphrases, since similar keyphrases we will be grouped together. Adding lemmatization as an option could be a great feature.

If the option is activated, the 'lemmatizer' component will be added to the spacy pipeline, and the lemma of words will be used instead of raw text to build keyphrases. There should also be a function to retrieve lemmatized documents. They will be built and stored during the pipeline process. This is necessary to calculate tf-idf.

I started a branch to build this feature : https://github.com/Logora/KeyphraseVectorizers/tree/use_lemmatizer

TimSchopf commented 2 years ago

Feel free to open a PR in the lemmatizer branch. I will then add this feature in a later release.

asmaier commented 10 months ago

Hello, anyone still working on this issue? I think this would be a great feature as in my tests I see a lot of keywords selected which for example are singular and plural of each other.