Enhancement: Remove Stopwords from '1000 Favourite Words'

[AlphaLaser]

A stop word is a commonly used word (such as “the”, “a”, “an”, or “in”) that doesn't add value to the feature.

Since they are not filtered, my favourite words for all users are these. Instead of this, we can filter out the stopwords and display the most used which would actually be something people would be interested in.

I'm not sure if JS/TS libraries for this task exist (They should). Otherwise, we can use Spacy with Python to make an API for the same.

import spacy

nlp = spacy.load("en_core_web_sm")
stop_words = nlp.Defaults.stop_words

def remove_stop_words(text):
    doc = nlp(text)
    return " ".join([token.text for token in doc if not token.is_stop])

text = "This is a sentence with stop words."
print(remove_stop_words(text))

Output:

sentence stop words.

[farkon00]

Even if such library does not exist, we can take the list of stopwords from the source code of the library.

[AlphaLaser]

True. These are all the stopwords spacy uses for reference:

# Stop words
STOP_WORDS = set(
    """
a about above across after afterwards again against all almost alone along
already also although always am among amongst amount an and another any anyhow
anyone anything anyway anywhere are around as at
back be became because become becomes becoming been before beforehand behind
being below beside besides between beyond both bottom but by
call can cannot ca could
did do does doing done down due during
each eight either eleven else elsewhere empty enough even ever every
everyone everything everywhere except
few fifteen fifty first five for former formerly forty four from front full
further
get give go
had has have he hence her here hereafter hereby herein hereupon hers herself
him himself his how however hundred
i if in indeed into is it its itself
keep
last latter latterly least less
just
made make many may me meanwhile might mine more moreover most mostly move much
must my myself
name namely neither never nevertheless next nine no nobody none noone nor not
nothing now nowhere
of off often on once one only onto or other others otherwise our ours ourselves
out over own
part per perhaps please put
quite
rather re really regarding
same say see seem seemed seeming seems serious several she should show side
since six sixty so some somehow someone something sometime sometimes somewhere
still such
take ten than that the their them themselves then thence there thereafter
thereby therefore therein thereupon these they third this those though three
through throughout thru thus to together too top toward towards twelve twenty
two
under until up unless upon us used using
various very very via was we well were what whatever when whence whenever where
whereafter whereas whereby wherein whereupon wherever whether which while
whither who whoever whole whom whose why will with within without would
yet you your yours yourself yourselves
""".split()
)

contractions = ["n't", "'d", "'ll", "'m", "'re", "'s", "'ve"]
STOP_WORDS.update(contractions)

for apostrophe in ["‘", "’"]:
    for stopword in contractions:
        STOP_WORDS.add(stopword.replace("'", apostrophe))

[arandomhooman]

Pls add this

[peterhanania]

to add a blacklist to the filter will cause even more lag to the site, as its already pretty heavy. I'll add a limit to words, only words with over 3 characters will show.