Open JoelNiklaus opened 3 days ago
Looks nice ! This is using external APIs and does not seem to have a PyPI package so we would need to implement it in Lighteval. This not not high priority but if you need it feel free to open a PR and we can help you set it up :)
Great, thanks! Yes, I see two avenues:
IMO option 1 is cleaner and also allows other people to use the metric more easily.
@chuandudx Would you be interested in taking this?
Issue encountered
The metrics only include rather outdated translation metrics.
Solution/Feature
Gemba MQM seems to be a current metric. Adding it would make translation evaluation better.