Reports: Ablation Study table

[InonS]

Motivation

Many research projects include an Ablation Study to compare model performance in the presence/absence of a combination of Boolean attributes such as the presence of certain model architecture elements, trainer configuration, etc. See examples in the "Related Discussion" section below.

This visualization is more effective than a Normal Form table of the specific values of each attribute.

Proposal Summary

Add a feature to Hyper-parameters Optimization (HPO) experiments (a.k.a. "sweeps") allowing each trial to be tagged (e.g. one-hot encoded) according to present/absent attributes, producing a table of the one-hot encoded attribute combinations with their respective score/s along with the Parallel Coordinates plot.

Related Discussion

1. https://www.baeldung.com/cs/ml-ablation-study
2. Example 1: Visual Question Answering (VQA) paper
3. Example 2: Deep Temporal Super-Resolution paper

[ainoam]

Thanks for bringing this up @InonS.

What interface did you consider for identifying the attributes for the ablation study within an HPO run?

[InonS]

Thanks for bringing this up @InonS.

What interface did you consider for identifying the attributes for the ablation study within an HPO run?

Please see the examples I shared here.

[ainoam]

@InonS

Just to verify we are aligned in terms: An HPO run goes through the combination of sets of values for a set of attributes in an effort to optimize a defined metric. An ablation study compares behaviour for attributes' presence/absence.

Are you looking for an interface like ClearML's HPO to automate an ablation study? Or, are you considering an added visualization for an HPO summary to create an ablation study out of an HPO run in the sense of grouping variations of each specific attribute?

Appreciate if you can provide some more details on what you have in mind.