The authors used uncertainty into NLP tasks and checked the improvement by using uncertainty.
2. What is amazing compared to previous studies?
They used uncertainty
measure the output confidence
enhance model performance
3. Where is the key to technologies and techniques?
They defined the total uncertainty is the sum of model and data uncertainty.
Model Uncertainty
It is defined the variance of expectation by using MC dropout.
Data Uncertainty
It is defined the expectation of variance.
The assumption that data uncertainty is dependent on the input.
4. How did validate it?
They tried 3 NLP tasks.
Sentiment Analysis
Task: classification
Datasets: Yelp 2013-2015, and IMDB.
Model: Baseline=CNN
The result shows that using Model and Data uncertainty improved model performance.
Especially, Model uncertainty is significant.
Named Entity Recognition
Task: sequence tagging(location, organization, person, and miscellaneous)
Datasets: CoNLL
Model: Baseline=LSTM
The result shows that using Data uncertainty improved score significantly.
Model uncertainty improved just 1%, because using MC dropout and chunk evaluation.
(The method and evaluation are not fit)
Language Modeling
Datasets: Penn Treebank(PTB)
Model: two-layer LSTM
The result shows that using both uncertainty improved score.
5. Is there a discussion?
above: The relationship between data uncertainty and NER entropy(lower is better).
below: The relationship between data uncertainty and NER tags.
These graphs show that data uncertainty can measure entropy and difficulty.
a1da4
commented
4 years ago
0. Paper
Quantifying Uncertainties in Natural Language Processing Tasks
1. What is it?
The authors used uncertainty into NLP tasks and checked the improvement by using uncertainty.
2. What is amazing compared to previous studies?
They used uncertainty
3. Where is the key to technologies and techniques?
They defined the total uncertainty is the sum of model and data uncertainty.
Model Uncertainty
It is defined the variance of expectation by using MC dropout.
Data Uncertainty
It is defined the expectation of variance. The assumption that data uncertainty is dependent on the input.
4. How did validate it?
They tried 3 NLP tasks.
Sentiment Analysis
The result shows that using Model and Data uncertainty improved model performance. Especially, Model uncertainty is significant.
Named Entity Recognition
The result shows that using Data uncertainty improved score significantly. Model uncertainty improved just 1%, because using MC dropout and chunk evaluation. (The method and evaluation are not fit)
Language Modeling
The result shows that using both uncertainty improved score.
5. Is there a discussion?
These graphs show that data uncertainty can measure entropy and difficulty.
6. Which paper should read next?
Analyze model uncertainty What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?