pytorch / ignite

High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.
https://pytorch-ignite.ai
BSD 3-Clause "New" or "Revised" License
4.51k stars 610 forks source link

To introduce `MetricGroup` class #3264

Closed sadra-barikbin closed 1 month ago

sadra-barikbin commented 1 month ago

🚀 Feature

Hi there! It might be useful to group some metrics together to have less code and more coherence. My use case was using some ignite metrics in HuggingFace Trainer API which expects a single callable for computing metrics. Having this feature, that could be easily realizable with Ignite. MetricGroup could be something like this:

from typing import Any, Dict
from ignite.metrics import Metric

class MetricGroup(Metric):
  _state_dict_all_req_keys = ('metrics',)

  def __init__(self, metrics:Dict[str, Metric]):
    self.metrics = metrics
    super(MetricGroup, self).__init__()

  def reset(self):
    for m in self.metrics.values():
      m.reset()

  def update(self, output):
    for m in self.metrics.values():
      m.update(m._output_transform(output))

  def compute(self) -> Dict[str, Any]:
    return {k: m.compute() for k,m in self.metrics.items()}

Usage:

metric = MetricGroup({'acc': Accuracy(), 'perplexity': Perplexity()})
metric.attach(engine) # or sth in HF Trainer API
vfdev-5 commented 1 month ago

OK, why not adding this.