`Unbabel/wmt22-comet-da` model not working as part of Huggingface evaluate

Unbabel / COMET

A Neural Framework for MT Evaluation

https://unbabel.github.io/COMET/html/index.html

Apache License 2.0

453 stars 72 forks source link

`Unbabel/wmt22-comet-da` model not working as part of Huggingface evaluate #125

Open alexandremuzio opened 1 year ago

alexandremuzio commented 1 year ago

🐛 Bug

I tried using Huggingface evaluate but it is not working with the newest Unbabel/wmt22-comet-da as part of unbabel-comet 2.0.0 release. I'm not sure if this is an issue with Huggingface evaluate lib or COMET.

To Reproduce

Please check this collab notebook.

Screenshots

The issue:

Environment

OS: Linux Version 2.0.0

ricardorei commented 1 year ago

It seems like Hugging face evaluate has to be updated to support recent comet versions. This is a bug on evaluate and not on COMET.

I'll try to find some time to investigate whats wrong with the current implementation of COMET in evaluate.

ricardorei commented 1 year ago

I created a pull request to evaluate to solve this: https://github.com/huggingface/evaluate/pull/443

Once it is merged, it should work.

Basically it has to do with an API change. You can see the difference here