Reproducing the precision-only results

ThomasScialom / QuestEval

MIT License

87 stars 26 forks source link

Reproducing the precision-only results #5

Open aagohary opened 2 years ago

aagohary commented 2 years ago

Hi, Thanks for the very interesting work. I am trying to reproduce the precision-only results in the QuestEval paper at EMNLP'21. Would you please provide instructions on how to do so? With the provided reference-less instructions, I am only getting correlation scores of ~29 on SummEval consistency and 14 on QAGS-XSum.

ThomasScialom commented 2 years ago

Hi Ahmed, thanks for your interest and sorry for the delay. What was your setup for QuestEval? Gonna look at it and compare it with mine. I see three source of disperency:

the data/correlation function etc.;
the parameters used to init QuestEval;
some recent changes in the code when I refactored it.

Note that I am working on a new version way faster with >0.5 coef on SummEval, which I plan to open-source maybe in January or February.