NDC Oslo lightning talk: Sami stopwords - How far have we gotten and why does it matter?

What I've sent in:

[...]

What are stopwords
What does the work with stopword-sami consist of
NRK as a text source and how to improve the stopword lists over time. Manual work, redlists and content crawling.
Solutions a stopword list can help you create: search engines, chatbots, plagiarism detection, sentiment analysis and other machine learning solutions.
Demo North Sami stopword list to show what simple linguistic understanding can do https://eklem.github.io/daq-proc/demo/document-processing/ (At least a beta version of North Sami stopword list will be added by August).
Show how far the lists for Lule Sami and South Sami has gotten and what's left to do. https://github.com/eklem/stopword-sami https://github.com/eklem/nrk-sapmi-crawler

[...]

Show what you get without stopwords:

With stopwords:

Smaller indexes (faster search and less storage need)
- amount of words in general
- typeahead (ngrams)
automatically generated keywords
Less noise in search result, more relevant hits.

Tell more about:

Why corpus of short texts are difficult to generate stopwords from (show the analysis on Norwegian based on Wikipedia.org/no)
Countering this with redlists
For search, the quality at the bottom doesn't need to be great as long as the volume has good quality, because that's what will be shown.

Illustrations to draw:

A lot of words/tokens
Quite some of those words being stopwords
Stopwords don't have a precise definition. It depends. A frequency sorted long array is then good. Cut off to fit your needs.
Smaller indexes, less noise at the bottom of a search result
Keyword generating/calculating doesn't need to be perfect since it's volume that bubles up.
Sami flag

eklem / stopword-sami