Comparing Distributions accross various runs.

[osoblanco]

🚀 Feature

I would like to be able to compare tracked distributions across several runs in order to gain insights about training/inference dynamics.

Motivation

This feature can allow seeing how a certain dynamic is changing within the model w.r.t. parameter changes across runs. For example: seeing the weight dynamics of the initial or final layer when the pretrained backbone model is changed or many many use cases akin to this.

Alternatives

You can always calculate the mean of the values and track that as a metric, but that robs you oh seeing the full granular dynamic

Additional context

These screenshots are from 2 different runs I tried comparing.

[gorarakelyan]

@osoblanco thanks for sharing the use-case, makes a lot of sense! We are going to release the Distributions Explorer in the coming releases. It would enable selecting, grouping and comparing tracked distributions.

A temporary workaround might be tracking those distributions as plotly figures and comparing them via Figures Explorer (released with Aim v3.13). Example:

[gorarakelyan]

@roubkar tagging you here so you are aware of the thread.