ML AL QS UncertaintySampling

[junxnone]

Uncertainty Sampling

Reference

• 主动学习（Active Learning）系列介绍（二）不确定度采样（Uncertainty Sampling）
• modAL - Uncertainty sampling

Brief

• 每次选择最不确定 batch data
• Input
  • Model predict Vector
• Output
  • Batch data index

不确定的评估标准

Name	Description	公式	Implement
Least Confident - 最小置信度	predict confidence越小越有价值		- alipy - modAL
Margin - 间隔	predict confidence 第一和第二的概率间隔越小越有价值		- alipy - modAL
Entropy	和所有 predict confidence 有关		- alipy - modAL

• NLP

Name	Description	公式	Implement
Token Entropy
Total Token Entropy
Sequence Entropy
N-best sequence entropy
MNLP - Maximum Normalized Log-Probality
MCLU - Multi-Class Level Uncertainty
BvSB - Best vs Second Best

Least Confident vs Margin vs Entropy

• 如图为三分类问题中三类不确定度计算方法的不确定度分布
• 颜色越深则越不确定 - uncertain，一般选取位于中间位置的示例作为query instance

不同的任务

• 分类
• 回归 - 最大方差

缺点

• 依赖于 model 的 predict value

History

• 1994 Least Confident - A sequential algorithm for training text classifiers
• 1994 Least Confident - Heterogeneous uncertainty sampling for supervised learning
• 2005 Least Confident - Reducing labeling effort for stuctured prediction tasks
• 2008 An analysis of active learning strategies for sequence labeling tasks
• 2001 Margin Sampling - Active hidden Markov models for information extraction - P309 ~ P318

[junxnone]

junxnone/aiwiki#118 junxnone/tech-io#798