Open Eastsidegunn opened 8 months ago
It will give great flexibility of metrics, but I think it doesn't have to implement in RAGchain. There are mainly two reason.
Rare F1
metric.)Plus, I think it will be too complicated to use our evaluator. Sometimes, framework should restrict flexibility for easy to use.
Ok,
I think that adding EM(Exactly match) metric is one of defined step. (conclusion of surfing on many benchmark)
Actually, I can't perceive setup which I can suggest conventinally. BLEU, and ROUGE score is wrapped official(maybe...? basically used) library... they have many variation according to n or perspective. this problem clearly need to be solved by our evaluator
In my think, evaluator should open folloing functions
how about metric_expaneded-version..? like metric.py, metric_expanded-version is collection of various metrics that are not officially(?) accepted name is tentative
We have few metric based on category of metric which can be controllable with few parameter (if based on n-gram, can choose n) more flexible!
in my opinion,
have more idea?