@inproceedings{dong-etal-2018-confidence,
title = "Confidence Modeling for Neural Semantic Parsing",
author = "Dong, Li and
Quirk, Chris and
Lapata, Mirella",
booktitle = "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2018",
address = "Melbourne, Australia",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/P18-1069",
doi = "10.18653/v1/P18-1069",
pages = "743--753",
}
1. What is it?
They used model, data, input uncertainty into semantic parsing.
2. What is amazing compared to previous studies?
They categorized the causes of uncertainty into
model
data
input
To estimate what we do not know.
3. Where is the key to technologies and techniques?
3.1 Model Uncertainty
There are 3 metrics
3.1.1 Dropout Pertubation
Use Monte Carlo Droput to estimate model uncertainty.
Calculate variance.
3.1.2 Gaussian Noise
Adding Gaussian noise, there are 2 methods.
Dropout was a Bernoulli distribution, but this noise is a Gaussian distribution.
After that, calculating variance as a dropout.
3.1.3 Posterior Probability
Use posterior probability.
3.2 Data Uncertainty
3.2.1 Probability of Input
They trained a Language Model on training data.
After that, calculate the probability of input.
3.2.2 Number of Unknown Tokens
Use a number of unknown tokens in the input.
3.3 Input Uncertainty
To measure uncertainty cause by ambiguous inputs, they used 2 metrics.
3.3.1 Variance of Top Candidates
Calculate top-k candidates variance.
3.3.2 Entropy of Decoding
Moreover, they used the sequence level entropy.
a' is sampled by using Monte Carlo sampling.
3.4 Confidence Scoreing
Using XGBoost regressor.
After that, wrapped by a logistic function.
4. How did validate it?
Task: semantic parsing
Data: Django, IFTTT
Table2: Model uncertainty is the best feature.
Table5: Dropout and Gaussian noise are important for scoring.
Figure3: The tradeoff between output proportion and F1score.
5. Is there a discussion?
6. Which paper should read next?
Use into Back Translation in Machine Translation.
Improving Back-Translation with Uncertainty-based Confidence Estimation
a1da4
commented
4 years ago
0. Paper
@inproceedings{dong-etal-2018-confidence, title = "Confidence Modeling for Neural Semantic Parsing", author = "Dong, Li and Quirk, Chris and Lapata, Mirella", booktitle = "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", month = jul, year = "2018", address = "Melbourne, Australia", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/P18-1069", doi = "10.18653/v1/P18-1069", pages = "743--753", }
1. What is it?
They used model, data, input uncertainty into semantic parsing.
2. What is amazing compared to previous studies?
They categorized the causes of uncertainty into
To estimate what we do not know.
3. Where is the key to technologies and techniques?
3.1 Model Uncertainty
There are 3 metrics
3.1.1 Dropout Pertubation
Use Monte Carlo Droput to estimate model uncertainty.
Calculate variance.
3.1.2 Gaussian Noise
Adding Gaussian noise, there are 2 methods. Dropout was a Bernoulli distribution, but this noise is a Gaussian distribution.
After that, calculating variance as a dropout.
3.1.3 Posterior Probability
Use posterior probability.
3.2 Data Uncertainty
3.2.1 Probability of Input
They trained a Language Model on training data. After that, calculate the probability of input.
3.2.2 Number of Unknown Tokens
Use a number of unknown tokens in the input.
3.3 Input Uncertainty
To measure uncertainty cause by ambiguous inputs, they used 2 metrics.
3.3.1 Variance of Top Candidates
Calculate top-k candidates variance.
3.3.2 Entropy of Decoding
Moreover, they used the sequence level entropy. a' is sampled by using Monte Carlo sampling.
3.4 Confidence Scoreing
Using XGBoost regressor. After that, wrapped by a logistic function.
4. How did validate it?
Task: semantic parsing Data: Django, IFTTT
Table2: Model uncertainty is the best feature.
Table5: Dropout and Gaussian noise are important for scoring.
Figure3: The tradeoff between output proportion and F1score.
5. Is there a discussion?
6. Which paper should read next?
Use into Back Translation in Machine Translation.