def ANLS(pred,answers): if answers[0] is not None: scores = [] for ans in answers: ed = editdistance.eval(ans.lower(),pred.lower()) NL = ed/max(len(ans),len(pred)) scores.append(1-NL if NL<0.5 else 0) return [max(scores)] return []
The ANLS smoothly captures the OCR mistakes applying a slight penalization in case of correct intended responses, but badly recognized. It also makes use of a threshold of value 0.5 that dictates whether the output of the metric will be the ANLS if its value is equal or bigger than 0.5 or 0 otherwise. The key point of this threshold is to determine if the answer has been correctly selected but not properly recognized, or on the contrary, the output is a wrong text selected from the options and given as an answer.
More formally, we define ANLS as follows:
where N is the total number of questions in the dataset, M is the total number of GT answers per question, a{ij} are the ground truth answers where i = {0, ..., N }, and j = {0, ..., M }, and o{q{i}} is the network’s answer for the ith question q{i} . N L(a{ij} , o{q{i}} ) is the normalized Levenshtein distance between the strings a{ij} and o{q{i}} (notice that the normalized Levenshtein distance is a value between 0 and 1). We define a threshold τ = 0.5 that penalizes metrics larger than this value, thus the final score will be 0 if the NL is larger than τ .
The intuition behind the threshold is that if an output has an edit distance of more than 0.5 to an answer, meaning getting half of the answer wrong, we reason that the output is the wrong text selected from the options as an answer. Otherwise, the metric has a smooth response that can gracefully capture errors in text recognition.
목표
DocVQA 대회에 대한 Metric 서칭
세부사항
기존 코드 사용
def ANLS(pred,answers): if answers[0] is not None: scores = [] for ans in answers: ed = editdistance.eval(ans.lower(),pred.lower()) NL = ed/max(len(ans),len(pred)) scores.append(1-NL if NL<0.5 else 0) return [max(scores)] return []
Reference
https://github.com/herobd/layoutlmv2/blob/main/eval_docvqa.py