69 pmi from 88 96 - Githubissues

Sotera / watchman

Watchman: An open-source social-media event-detection system

GNU General Public License v2.0

20 stars 7 forks source link

69 pmi from 88 96 #124

Open drJAGartner opened 7 years ago

drJAGartner commented 7 years ago

[x] Create passthrough featurizer
[x] Create Clusterer
[x] Count individual words
[x] Create & count sorted word pair
[x] Calculate PMI: https://en.wikipedia.org/wiki/Pointwise_mutual_information#Definition
[x] Create clusters with likely/unlikely score
[ ] Filter out nodes with unlikely score during community building
[ ] Rewrite jobs to pass correct varriables & not use old image/text clusterer.

drJAGartner commented 7 years ago

Hey Guys, I got the unit testing for this done. It runs somewhat slowly, and I think creating the individual word list and word pairs list during featurizing could help. If you wanted to finish off the last bits, I'd be great with that.