aronwc
commented
11 years ago - Active Learning for Imbalanced Sentiment Classification
- To select a new example to be annotated:
- Select K most confident positive and negative examples (call this set S_pos, S_neg)
- Select least confident element of S_pos and S_neg (call them x_pos, x_neg)
- Human labels x_pos, machine labels x_neg
- repeat
- Two classifiers are built by training on different feature subsets: one for selecting most confident, the other for least confident.
- The relative feature set sizes has big impact: most confident classifier should use fewer features (~1/16-1/8)
- Under-sampling is used when training the classifier (to deal with class imbalance)
- Reducing class imbalance during active learning for named entity annotation
- Sometimes called "boosted disagreement"
- Active learning ranking function has a parameter to boost weight for minority class.
- Why Label when you can Search?
- "Guided Learning:" user searches for an equal distribution of positive and negative examples
- This is often much better than active learning... _...but the human cost may be greater (must search for _and* label examples)
- A hybrid strategy can work well:
- Given a predefined cost factor (e.g., guided learning is 8 times costlier than active learning), compute a cost-benefit tradeoff to determine when to switch from guided to active learning
- "The certainty of the underlying model that the as-of-yet unexplored disjuncts are of the wrong class hinders
the success of the active learning ". Translation: when the minority class consists of multiple clusters, finding those different clusters quickly is important. Here's an interesting plot of this, where different clusters of blue dots represent subspaces of the minority class. Bumps in accuracy seem to coincide with discovering these subspaces.
reidpr