Closed NISH1001 closed 1 year ago
Re: There might be a small bug in evalem.evaluators.Evaluator
that returns a dictionary mapping from MetricName -> Dict result. Bug is, if we have 2 metric initialized from JuryBasedMetric
, the mapping only returns the latest result as both metric would have same class name. Might be better if there could be counter or something (a static variable within JuryBasedMetric class)
Major Changes
This PR adds a bunch of new semantic metrics which are plug-and-play with existing
evalem.evaluators.Evaluator
and pipelines.evalem.metrics.BleuMetric
for bleu scoreevalem.metrics.MeteorMetric
evalem.metrics.RougeMetric
evalem.metrics.SacreBleuMetric
Minor Changes
Now, we make use of
model_params
andeval_params
throughkwargs
inevalem.pipelines.SimpleEvaluationPipeline.run(...)
method to pass through corresponding forward pass for model wrapper and evaluators.The
evalem.metrics.semantics.SemanticMetric
now only inherits fromMetric. Previously it was inherited from
JuryBasedMetric`. So, now, we use multiple inheritance to derive existing metrics for semantic scores. This allows us to have type hierarchy that can loosely segregate from Jury backend.