Useful abstraction for metrics tracker metrics

[rajs96]

🚀 Feature

Integrating timeseries metrics. We can create an abstraction, or class, that should contain the metrics recorded in a time series.

Motivation

At work, we've often been interested in seeing how metrics besides the loss change throughout the training process - using a designated object with built-in functionalities would save the time of having to implement these functionalities ourselves. For example, we may want to plot metric values throughout training, return the max metric throughout the training for each metric, average timeseries metrics from multiple models for cross validation, etc.

Pitch

We should create a class that makes it easy to add timesteps of Metric modules (or MetricCollections), and add in useful functionalities for the class. Refer to the following pseudocode:

    ts = TimeSeriesMetrics(MetricCollection(Accuracy(), Precision(), Recall(), F1())
    for _ in range(epochs):
        model.train_epoch()
        preds = model(data)
        ts.add_timestep(preds, targets)

    # get best f1 score and the associated timestep
    best_timestep, best_f1 = ts.best_metric('f1', return_timestep=True)

    # get all precision scores in order
    precision_scores = ts.all_metrics('precision')

    # given multiple timeseries metric objects, average along the timeseries axis
    # for example, if you were doing cross validation
    ts_metric_avg = timeseries_avg([TimeSeriesMetrics1, ...])

Alternatives

We could theoretically just have the user maintain a Python List of Metric Modules (or dicts computed from these modules) and implement functions to act on these lists, rather than creating a class.

Additional context

I came across these problems / ideas when training transformer models for 15-20 epochs with cross validation.

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[Borda]

@rajs96, it sounds a great addition to this package, are you interested in implantation this? :rabbit: it would be a highlight for the next reease! cc: @PyTorchLightning/core-metrics

[rajs96]

@rajs96, it sounds a great addition to this package, are you interested in implantation this? 🐰 it would be a highlight for the next reease! cc: @PyTorchLightning/core-metrics

Yup - what are next steps then?

[Borda]

I would say make an abstract/base class for this metric group and then we will particular metrics in separate PRs... btw, ping us on SLack so we can easier coordinate :]

[Borda]

@rajs96 how is it going, do you think we can make some initial draft about this week?

[rajs96]

@rajs96 how is it going, do you think we can make some initial draft about this week?

I can start working on a PR draft this weekend - would that work?

[Borda]

I can start working on a PR draft this weekend - would that work?

Sooner is better so we have more time to fine tuning it =)