Regarding metric chrf's implementation

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.

https://www.eleuther.ai

MIT License

6.9k stars 1.84k forks source link

Regarding metric chrf's implementation #2256

Open PinzhenChen opened 2 months ago

PinzhenChen commented 2 months ago

Hi,

In this repo, the chrf metric implementation calls sacrebleu.corpus_chrf() with default parameters: character order 6 and word order 0. Perhaps in metric.py it would be nice to include those two optional parameters so users can specify this in task.yml, for example to facilitate chrf++ which uses word order 2?

Thanks!

baberabb commented 2 months ago

Hi! That sounds reasonable. Would you like to open a PR?

JINO-ROHIT commented 3 weeks ago

Hi can i take this up?

PinzhenChen commented 3 weeks ago

Hi @JINO-ROHIT , yes sure, I have not implemented this.