Thank you very much for making your implementation available. I am trying to reproduce the results for Shannon score and Information Difference score using the SummEval benchmark. However, the results I am getting are really poor compared to the paper's (all correlations < .3). I am running the 1.6k source texts and their summaries through the class below.
Dear authors,
Thank you very much for making your implementation available. I am trying to reproduce the results for Shannon score and Information Difference score using the SummEval benchmark. However, the results I am getting are really poor compared to the paper's (all correlations < .3). I am running the 1.6k source texts and their summaries through the class below.
Am I doing anything wrong? Thank you!