search

Y-IAB / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
0 stars 0 forks source link

Add fragma based model & add xcomet/wmt23-cometkiwi metrics #9

Closed myeongho-jeong-yanolja closed 7 months ago

myeongho-jeong-yanolja commented 7 months ago
  1. Add fragma API based model for chat-completion (currently, for gemini model). However, it is not tested because our server can't access to fragma API 😢 .
  2. Add xcomet / cometkiwi23 metrics for reference-free translation evaluation. It is almost similar with cometkiwi22, but with another pretrained model.

CAUTION: Because this repository uses various model-based metrics, you need to allocate cuda device for Bartscore/cometkiwi/xcomet for yourself!