Shreyanand
commented
1 year ago BARTscore: https://arxiv.org/pdf/2106.11520.pdf ROUGE: https://aclanthology.org/W04-1013.pdf BLEU, BERTScore, METEOR, WMD Blog to read: https://blog.paperspace.com/automated-metrics-for-evaluating-generated-text/ Metrics Survey: https://arxiv.org/pdf/2006.14799.pdf , https://aclanthology.org/2021.triton-1.6.pdf Medium Implementation blog: https://medium.com/@vincentchen0110/evaluating-your-text-generation-results-simple-as-that-e74547383181 Human performance ratings based on chatGPT style architecture (good read but may not be applicable): https://arxiv.org/pdf/2203.02155.pdf?fbclid=IwAR2nZdBpdZZzvxpwI6H_bRmP4RwGOyzke9Ud63lWBe1YlyI_1BRAFhnUMUg Haystack and Deepset implementation of QA framework: https://www.deepset.ai/blog/how-to-evaluate-question-answering They use EM, F1, and SAS metrics (F1 here is on the overlap of bag of words of predicted and ground truth so it is more forgiving than Exact match)
suppathak
codificat
In order to compare different QA models for user documentation, we will have to decide on quantitative metrics that evaluate the models. As a part of this issue add resources and discussion points for the metrics to be used. Add a notebook that show some of these metrics.