gomate-community / rageval

Evaluation tools for Retrieval-augmented Generation (RAG) methods.
Apache License 2.0
81 stars 9 forks source link

enhance base metric #45

Open faneshion opened 4 months ago

faneshion commented 4 months ago

Enhance the base metric for robust evaluation:

  1. add attribute: self.task to distinguish input format.
  2. add function: def _validate_data(self, input: Dataset) -> bool to check the validity of the data.
faneshion commented 4 months ago

The task will be removed outside of the Metric. We can divide all metrics into four categories: AnswerCorrectness, AnswerGroundedness, ContextRelevancy, and ContextAdequacy.

For each metric, we can decorate it to add the metric_type attribute:

def add_attribute(attribute_name, attribute_value):
    def decorator(cls):
        setattr(cls, attribute_name, attribute_value)
        return cls
    return decorator

# 使用装饰器定义类的属性
@add_attribute('mtype', 'AnswerGroundedness')
class _em_answer(Metric):
    pass

# 创建类的实例并访问属性
metric = rl.metrics._em_answer()
print(metric.mtype)  # 输出: AnswerGroundedness