As an end user, I want to be able to clean my workset

[cwulfman]

See:

• 33

• 34

• 35

• 36

• 37

On the dashboard is a checkbox menu of eight choices:

1. Apply Stopwords
2. Make Lowercas 3. Lemmatize 4. Stem 5. Search and Replace
3. Page Features
4. POS Tags
5. Token Count Limits

As for the fltering options, the dashboard does not have an "Apply" button, so the front end, to be idempotent, needs to send a PUT request containing the current set of selected cleaners each time one of the buttons is pressed.

Some of these operations (like Stem and Lemmatize) are standard NLP operations on tokens; others (like POS Tags and Token Count Limits) seem to be filters (e.g., include in the full token list only tokens that are Nouns; include in the token list only tokens that occur more that ten times in the dataset; exclude all tokens that appear in a stop list). Others, like Make Lowercase and Search and Replace, are basic string operations, but they alter the token data.

"Page Features" refers to the header, footer, and body sections in the EF data; users should be able to choose which of these sections to include in their analyses.

To implement this feature, it will be prudent to break the task down into individual sub-features, by category:

• filters (Apply Stopwords, POS Tags, and Token Count Limits, Page Features). These are just like the operations in the Filters widget; they are added or subtracted from the filter chain. They are pure functions: they are indempotent and free of side-effects, so they do not entail modifying the data.
• transformers (Make Lowercase; Lemmatize; Stem; Search and Replace) are very different; they mutate (modify) the data: applying one of these operations yields a new set of new data, which must be managed. These transformers are of two types:
  1. string operators: these are operations that are applied to strings of characters: convert A to a; B to b, and so forth; make context-free substitutions (i.e., /colour/color/, where the only context is the string value of the token). These basic string operators are part of most programming languages (including Python).
  2. token operators: In NLP, the term /token/ usually refers to /word tokens/ -- what we usually think of as words. Stemming and lemmatization are syntactic operations, and they require programs and databases to execute. Open-source stemmers and lemmatizers exist, but using them adds another layer of complexity to torchlite. They also assume the data is clean (i.e., that all the tokens are actual words and not dirty-OCR noise)

suggestion: a noise filter would probably be a useful thing : a filter that used a dictionary (or dictionaries, in the case of multilingual corpora) to throw out token-strings that are not found in the dictionary.

[jswatsch]

HTRC discussed and want to include the following choices:

Apply Stopwords Make Lowercase Lemmatize Stem Search and Replace Page Features POS Tags Token Count Limits

Clarification on "page features". Page features is a selector between header, body, and footer.

No noise filter at this time.