I have been trying to test several models on the GEM-benchmark metrics. I followed the tutorial provided both on GitHub and the official website and have been able to generate a submission file with generations and GEM-ID keys. However, when I attempt to generate output scores, I notice that I am missing several scores I wish to have.
For example, in the requirements.txt file, the package of rouge-score is included, however, the output scores do not contain any rouge metric. Furthermore, I attempted several times to generate output scores with --heavy-metric flag, however, this is always skipped. Regardless of whether I include the flag or leave it out, the same metrics are returned.
I attached an example of my output scores below:
An example of the generation is shown below here:
More information:
I cloned the repo in my google drive, cd'd in the file and pip installed both the normal requirements file as heavy requirements. I did this several times ensuring that everything was installed
I also tried generating the metrics by manually choosing the metrics with --metric-list, but that did not work either
I attempted to pip import gem_metrics, but this did not resolve my issues.
Hi,
I have been trying to test several models on the GEM-benchmark metrics. I followed the tutorial provided both on GitHub and the official website and have been able to generate a submission file with generations and GEM-ID keys. However, when I attempt to generate output scores, I notice that I am missing several scores I wish to have.
For example, in the requirements.txt file, the package of rouge-score is included, however, the output scores do not contain any rouge metric. Furthermore, I attempted several times to generate output scores with --heavy-metric flag, however, this is always skipped. Regardless of whether I include the flag or leave it out, the same metrics are returned.
I attached an example of my output scores below:![afbeelding](https://user-images.githubusercontent.com/63370145/221655660-2314c0c4-fd12-41d5-9300-a60c45827d17.png)
An example of the generation is shown below here:![afbeelding](https://user-images.githubusercontent.com/63370145/221656092-ac581934-3274-4369-99f1-9438a044cc2f.png)
More information:
Could someone help uncover what I am doing wrong?
Kind regards