NASA-IMPACT / evalem

An evaluation framework for your large model pipelines
0 stars 0 forks source link

[alpha] Addition of more semantic metrics #12

Closed NISH1001 closed 1 year ago

NISH1001 commented 1 year ago

Major Changes

This PR adds a bunch of new semantic metrics which are plug-and-play with existing evalem.evaluators.Evaluator and pipelines.

Note: We can always use evalem.metrics.JuryBasedMetric class to have more. These 4 implementations exist to make it more verbose within the pipeline abstraction.

Minor Changes

NISH1001 commented 1 year ago

Re: There might be a small bug in evalem.evaluators.Evaluator that returns a dictionary mapping from MetricName -> Dict result. Bug is, if we have 2 metric initialized from JuryBasedMetric, the mapping only returns the latest result as both metric would have same class name. Might be better if there could be counter or something (a static variable within JuryBasedMetric class)