search

NASA-IMPACT / evalem

An evaluation framework for your large model pipelines
0 stars 0 forks source link

add CV metrics to evalem #28

Open NISH1001 opened 1 year ago

NISH1001 commented 1 year ago

cc: @rbavery @weiji14

I looked at torchmetrics. And it seems since the metric there -- just like evalem.metrics -- also uses (predictions, references) type of argument, I think we can have new wrappers around torchmetrics at evalem.cv.metrics

image

For instance, the base metrics in the evalem has evalem._base.metrics.compute(...) method to be overridden by downstream implementation: https://github.com/NASA-IMPACT/evalem/blob/main/evalem/_base/metrics.py#L38

One evalem.nlp.metrics.ExactMatchMetric has the implementation for example: https://github.com/NASA-IMPACT/evalem/blob/main/evalem/nlp/metrics/basics.py#L16

NISH1001 commented 1 year ago

Related thread: https://github.com/NASA-IMPACT/evalem/issues/19