Open DhananjayAshok opened 1 month ago
Hi, DhananjayAshok, thanks for your issue. I suppose the error is because that each "golden_answer" has several references and they are split with '/', which is also mentioned in the Data Description part of readme. You can check the data. We will make it more clear.
Hi there, I am running the evaluation metrics of Lexical Match and BertScore. But I am getting extremely different results from those found in the paper. I wanted to bring this to your attention to make sure I am not making any mistakes in the code and set up.
I start by loading the EVOUNA benchmark and combining the points into a single dataframe such that each row is an example of its own (with a gold answer and prediction from some model)
The above code gives me a dataframe with columns: ["dataset_name", "question_id", "question", "prediction", "reference", "judge", "model"] and has (after dropping NaNs) 25358 rows.
I loop over the dataframe and compute the scores in the following ways:
This tries to follow the lexicalMatch description outlined in the paper. With this, I get very different results from the original paper. I get a bertscore correlation of:
The paper uses f1 with a threshold of 0.5, but if I use this the accuracy is less than 20%
Similarly the lexical match metric shows an accuracy of 41% which is around half of the amount shown in the paper.
Could the evaluation code be released so that I can understand what my mistake is here?