Question Regarding Optimal MIA and Overall Desired Objective

[CerebralSeed]

What is considered optimal for the MIA score?

Obviously, it should be lower than the initial model. But just wanted to clarify, are we aiming to have the last chart with as much overlap as possible between the Test and Forget set and a high overall score on the test set, or would an MIA score of less than 0.5 be ideal(and, yes, I can get this significantly lower than 0.5)?

Just trying to clarify the metrics which will be considered an "improved" result.

[CerebralSeed]

Perhaps my question was a bit vague.

Here is one example where the model is intentionally trained to be LESS than a 0.5 MIA score, but has a very different looking histogram with not much overlap.

So would the scoring of this challenge be seeking to weight results based on:

1. Some combination of lowest raw MIA score and highest test accuracy; OR
2. An MIA score around 0.5 with similar overlaps between Test and Forget sets and highest test accuracy.

[CerebralSeed]

I think I found my answer in the FAQ here: https://unlearning-challenge.github.io/faq/

Basically, it reads to me like you want the test and forget MIA histograms to be virtually identical with as high of test set accuracy as possible.

[eleniTriantafillou]

Hi - thanks for your question, and apologies for the delayed response.

Generally, we assume that the ideal unlearning algorithm is one that best "matches" the model that we would have gotten had we retrained from scratch using only the retain set (i.e. without the forget set). As such, the ideal score for the MIA is the score that that ideal unlearning algorithm would obtain.

Note that, in some cases, the ideal MIA will be 50%. But there are certain factors (that don't necessarily have anything to do with unlearning quality) that may cause even the ideal retraining-from-scratch to not have an MIA of 50% (e.g. if the forget and test sets have different distributions). So generally, one should use the reference point of the retrain-from-scratch unlearning method for evaluation. We had just updated our notebook to include that reference point (please see the last section).

Also, please note that this simple MIA is provided for demonstration purposes in the notebook. In the actual competition, we will use different metrics.