Add metric logging

[Dref360]

Summary:

Add some utils to keep track of metrics over time.

This is WIP, but what do you think of this API? It would be quite simple for the users to get a metrics for each dataset size.

We can also add utilities for that such as MetricMixin.get_active_metric(metric_name="precision") : Dict[int, float]

Features:

Fixes #220

Checklist:

• [ ] Your code is documented (To validate this, add your module to tests/documentation_test.py).
• [ ] Your code is tested with unit tests.
• [ ] You moved your Issue to the PR state.

[parmidaatg]

I like this. thanks a lot @Dref360. I was only wondering don't you think it is more descriptive to have "active_step" as the key for metrics and have "datasize" as a field ?

[GeorgePearse]

This looks great, will give it a try as soon as it's ready.

[Dref360]

I like this. thanks a lot @Dref360. I was only wondering don't you think it is more descriptive to have "active_step" as the key for metrics and have "datasize" as a field ?

Good idea, not sure how to keep track of the active step tho. Do you have an idea? Like just a field in ModelWrapper?

[Dref360]

I think this is ready for review. I added the dataset size in the response of get_metrics.

[rafapi]

Is there an entry in the docs for this? Regardless of that, this looks good!

[Dref360]

@rafapi just added an example in our documentation.

[Dref360]

Quick video showcasing the feature. https://www.loom.com/share/4b7d0eeee187463cac12ecd57828c5fe