Metrics support

[zacps]

As a convenience utility we should provide the ability for users to log numeric metrics during their experiment.

These metrics should be cached along with other results.

By narrowing what metrics we support we can automatically provide aggregation (median, mean, stderr, ...) calculations and potentially provide (live?) automatic plots (via matplotlib/seaborn?).

[Liam-Scott-Russell]

@zacps I'd be interested in picking this up. Could you provide some more details maybe?

Are we looking at some logger method that a user can use (kind of like print())?

def my_code():
    memento.logger.log("starting experiment")
    # Other stuff
    memento.logger.log(f"got a result: {result}")

[zacps]

More like a store for time-series values. For example if you wanted to record the loss of a model during training.

Really all this has to be is a thin wrapper around pandas. Primarily we should take care of:

• Saving metrics, both for checkpoints and caching their final results
• Potentially calculating mean/median/stderr from repeated runs automatically
• Potentially making it easy to turn collection of specific metrics on or off, as some might be expensive to collect. This could possibly work by the user specifying closures to calculate each metric, then a configuration key (from the configuration matrix probably) enabling/disabling each of them
• IIRC pandas dataframes are immutable so we should record them in some other structure and convert them at the end