Open szutenberg opened 1 year ago
This issue is caused due to some randomness in rouge score code (in evaluate repo) and I fixed it by setting numpy random seed in the script. Please take a look here
I treat this fix as WA because now indeed it's deterministic but I feel that it's just way of hiding the problem.
Are you able to explain where do we have indeterminism in rouge score calculation? Aren't these scores just averages from all examples? Do you know how are they being calculated? Thanks!
Ideally, they should be deterministic as they are F-1 scores of different n-grams. I'm looking at an existing issue in their repo and will update once I test the actual fix
I found this issue here that talks about the same problem. They enable the BootstrapAggregator by default in the code which does random sampling to compute confidence intervals which causes run-to-run variation in ROUGE scores. From what they mention in the issue, it can be disabled safely. I've tested it and setting use_aggregator=False produces deterministic results. I've created a PR for the same
We found that evaluation.py is not deterministic.
I narrowed down to small and fast reproducer using 100 examples which are already decoded.
Reproducer code:
Results from 8 runs:
Differences are larger than 1% (15.2326 vs 15.0713 in rouge2) which makes this tool a bit problematic for robust accuracy evaluation.
Required files: preds_decoded_text.txt target_required.txt
I ran my experiments on docker ubuntu:latest to make sure that this is not machine/environment issue. Preparing environment:
Pip freeze: pip_freeze.txt