Figure 1

[dilyabareeva]

We need a nice-looking, attention-grabbing Figure 1 that encapsulates the idea of our library. Some features it might have:

• examples of top-k attribution images for a specific sample (has to be a cute sample)?
• comparison plot of different downstream evaluators?
• performance of an explainer across different datasets?
• accuracy plots for label poisoning (cite some papers here)?
• leave-k-out correlation plots (e.g. x-axis=k, y-axis=correlation)? some other statistics?

[gumityolcu]

What do you think about this kind of classification of benchmarks for Quanda:

Leave-k-out Counterfactuals:

• No metrics yet. Maybe 1 or 2 can be quickly implemented?

ML Related Tasks:

• Mislabeled Data Detection
• Dataset Cleaning
• Class Prediction
• Subclass Prediction

Input Dependence Sanity Checks:

• TopKOverlap : Measures how much the explanations depend on the test point (some methods may tend to produce similar ranking of training points due to using the model parameters). The overlap of top-k training datapoints of different test points should be low.
• Model Randomization test: Measures explanations' dependence on the model

This would correspond to grouping Model Randomization and TopKOverlap because they test dependence on the inputs of the explanations scheme. Then we group dataset cleaning with the "localization" benchmarks and call them "ML tasks". Probably, there is a better name for it.

These could be Figures 1.a,b and c. We can give spyder plots/mislabeling detection curves/tables in these subfigures. Different datasets can be color coded. And these subfigures don't need to be presented in the standard academic fashion. Maybe there is a better way to organize them. They could also possibly be incorporated into a single figure. I just think this division makes sense for the paper.

[gumityolcu]

unrelated idea: Top-k-overlap and top-k-localization metrics also generate different values as k changes. Maybe for these benchmarks, we can generate a plot with a line for each explainer as k=1,2,3,4,...

[dilyabareeva]

Inspiration: https://github.com/MadryLab/trak/blob/main/assets/imagenet_figure.jpeg