Reduce the number of tags - Githubissues

ptypes-nlesc / stereotype-map

Mapping videos and predefined stereotypes using word embeddings

https://stereotype-map.readthedocs.io/en/latest/

Apache License 2.0

0 stars 0 forks source link

Reduce the number of tags #4

Open lyashevska opened 9 months ago

lyashevska commented 9 months ago

The following strategies can be used:

[x] Remove uncommon tags with low number of occurrences (e.g. 1000, but depending on the size of the dataset and domain)
[ ] Aggregate tags that refer to the same term: 1) group tags that differ only by special characters, e.g. "-", " ", etc. 2) group morphologically similar tags (e.g. singular and plural terms, different verb tenses).
[x] Remove irrelevant (e.g. HD Porn, etc) tags.
[ ] Use Porter's algorithm, lemmatization or stemming
[x] ptypes-nlesc/ptypes-topic-modeling#12

lyashevska commented 9 months ago

See Wang et al 2015