Research what kinds of features we might engineer

[divilian]

We already know that "whether the document has stem X somewhere in it or not" is a usable feature. What about all kinds of other things?

• length of thread?
• number of different voices in thread?
• average length of comment in thread?
• does thread have spelling erors?
• does thread have swear words? Etc. Some of this you will get by sitting in a quiet room and brainstorming. Others you will get by Googling "text mining classification features" and also looking at the papers which are doing classification of texts.

[akochans]

For more info, I put a google doc in our shared google drive folder. Most bolded citations in the document are also in Zotero.

Schmidt and Wiegand 2017 talk of "simple surface" features, this is most commonly n-grams, and while effective, they are rarely used alone. Additional simple surface features: Linguistic features from Nobata et al 2016:

• length of comment in tokens
• average length of word
• number of punctuations
• number of periods, question marks, quotes, and repeated punctuation
• number of one letter tokens
• number of capitalized letters
• number of URLS
• number of tokens with non-alpha characters in the middle
• number of discourse connectives, based on [13]
• number of politeness words
• number of modal words (to measure hedging and confidence by speaker)
• number of unknown words as compared to a dictionary of English words (meant to measure uniqueness and any misspellings)
• number of insult and hate blacklist words Syntactic features from Nobata et al 2016:
• parent of node
• grandparent of node
• POS of parent
• POS of grandparent
• tuple consisting of the word, parent and grandparent
• children of node [8]
• tuples consisting of the permutations of the word or its POS, the dependency label connecting the word to its parent, and the parent or its POS Distributional semantic features from Nobata et al 2016:
• three types of embedding-derived features. (first two are based on averaging the word embeddings of all words in the comment)
  • In one we use a pre-trained embeddings .
  • In the second we use word2vec10 to train embeddings from our large corpora
  • For our third embeddings features, we develop a comment embeddings approach akin to [8].

[divilian]

For the "words not in the dictionary" feature, what might be the best dictionary to use?

[divilian]

Mostly done. @akochans's Google Doc has a roughly-prioritized list of features we want to try (boldface = high priority; red = our own invention instead of Nobata's.) As for "best dictionary" for judging non-real-words, there seems to be no consensus for this.