aatmanvaidya
commented
11 months ago Previous Work on the Umbrella Issue
- regex/string matching
- capturing common word mis-spelling.
Derive Root Words of the words in the Slur List
From what I broadly understand we want to do - Lemmatization and Stemming The root word is called - Lemma
Automated/ Semi-Automated Approaches
Lemmatization for English
- https://www.geeksforgeeks.org/python-lemmatization-approaches-with-examples/
- Popular python libraries -> WordNet, WordNet (with POS tag), TextBlob, TextBlob (with POS tag), spaCy, TreeTagger, Pattern Gensim, Stanford CoreNLP etc
- Comparing NLTK, TextBlob, spaCy, Pattern and Stanford CoreNLP - Link
- https://www.geeksforgeeks.org/python-lemmatization-with-nltk/
- Stemming and types of stemming's - https://www.engati.com/glossary/stemming#:~:text=popular%20stemming%20algorithms%3F-,What%20is%20stemming%20in%20NLP%3F,reduced%20to%20a%20standard%20form.
- Porter's algorithm (suffix stripping) - https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html
- https://stackoverflow.com/questions/32520521/detect-words-with-the-same-root
Lemmatization for Hindi
- a simple stemmer in java - https://stackoverflow.com/questions/4007558/is-there-is-any-stemmer-available-for-indian-language
- NLTK library
- A Hybrid Approach for Lemmatizer for Hindi Language
- A Rule-based Part-of-Speech and Morphological Tagging Toolkit - Link1, Link2
- Multilingual NLTK for POS Tagging and Lemmatizer
- Taxonomic survey of resources for Hindi Language NLP systems ->See section
4.2.9 - https://paperswithcode.com/task/lemmatization
Approaches with Human Intervention
- Rule-based Lemmatization
- Manual Annotations of
RootandNot Root
github-actions[bot]
We are using the phrase "root word" as a catchall term. There are many words that are misspellings of a common slur. For instance fuck could be spelled as fck, fk, fcuk etc.
This was highlighted as a problem during our annotation sprints too. Annotators weren't sure if they should annotate all the misspellings or would it be ok to just annotate the root word and our system will understand that those annotations are valid for all derived words.
In scope for this issue is,