What is the reason not to compare with MIA with "Retrained" model?

[NorthStar]

Hello I notice the baseline simple MI diagram looks like a "decrease of MIA from pre-trained model to fine-tuned model", which is different from what we consider privacy baseline, where the goal is -- or I think should be -- the MIA of retrained model that does not include the offending data.

Image below:

Am I confused, or is this a developmental area?

Thanks!

[eleniTriantafillou]

Hi Mimee,

Thank you for your question!

I certainly agree with you that the oracle "reference point" would be how well the retrained model (trained without the forget set) is defending the MIA; ideally, the unlearned model should defend MIAs equally well as that reference point. This figure is simply a sanity check: we're showing that the attacker is more successful against the "original model" (before any unlearning is done) compared to the unlearned model; and that the unlearned model's distributions of forget and test losses are more "similar" to each other compared to how similar those two distributions of losses are within the "original model". I agree with you that this isn't saying much (having an unlearning algorithm that does better than no unlearning at all is a very low bar, of course :)). We use this figure for illustration purposes.

(Note as well that the MIA we use here is not a particularly strong one, it's simply used as an example, so conclusions drawn from it should be perhaps taken with a grain of salt anyway).

Hope this helps! Eleni

On Mon, Jul 3, 2023 at 6:14 PM Mimee @.***> wrote:

Hello I notice the baseline simple MI diagram looks like a "decrease of MIA from pre-trained model to fine-tuned model", which is different from what we consider privacy baseline, where the goal is -- or I think should be -- the MIA of retrained model that does not include the offending data.

Image below:

[image: image] https://user-images.githubusercontent.com/1208951/250626054-28e53fcd-fe94-41ed-b419-f618f390e1a4.png

Am I confused, or is this a developmental area?

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/unlearning-challenge/starting-kit/issues/6, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3R53LWGYFGMTY2HOPYNV3XOL4XXANCNFSM6AAAAAAZ4W3244 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[NorthStar]

Thank you, Eleni, for the response. Regarding my original question, I take it to mean that it is still an active developmental area.

The notebook code seems misleading, while in reality the entire competition is simply not ready.

Do you have a timeline regarding the completion of the competition's starting kit? The blogposts make it seem like it can be worked on already.

For a complete starting kit, I would expect clarifications on the following:

• what is the input/output - do the datasets necessarily constitute as classification-only? Are CIFAR images the benchmark?
• what is implemented that constitutes as an "unlearning" algorithm? Is it the function call unlearning(net, retain, forget, validation), do the networks need to be pre-trained? Is there access to the original training methods?
• what is being evaluated - imo privacy is to be evaluated on the models trained without forget_set, yet there is no code path for that in the notebook. Related to #9 .

Please let me know if you need help.

Thanks, Mimee

[eleniTriantafillou]

Hi Mimee,

Please refer to our website for info (e.g. timelines) on the competition: https://unlearning-challenge.github.io/ The competition hasn't yet started (we're aiming to launch in mid July) and please note that the purpose of the starting kit is to provide a simple illustration of the unlearning problem and a simple way of tackling it. There is also an FAQ page on our website that you might find helpful: https://unlearning-challenge.github.io/faq/

To answer your questions specifically:

• Re: input-output: No, CIFAR-10 is a simple example we are using in the starting kit. The competition will be run on natural images of faces (we won't release those datasets). The model is an age classifier and the task is to forget a subset of the training set. So yes, we will consider only classification tasks.
• Re: the unlearning algorithm: indeed, the function signature is as you mention: unlearning(net, retain, forget, validation). The first argument is a pretrained model (the "original model") that is trained on the training set. The retain and forget sets (second and third arguments) represent a particular partition of the original model's training set; the goal is to forget the latter. We also give a validation set of held-out data, as the final argument, that the original model was not trained on.
• Re: evaluation, we won't reveal our metrics until after the competition. Please see the FAQ for a more detailed answer on that. But your intuition is correct that the ideal unlearning algorithm is to retrain the model from scratch without the forget set. The goal of the competition is to incentivize the development of models that are similar to that, but much more computationally efficient.

Hope this helps.

-Eleni

On Wed, Jul 5, 2023 at 8:42 PM Mimee @.***> wrote:

Thank you, Eleni, for the response. Regarding my original question, I take it to mean that it is still an active developmental area.

The notebook code seems misleading, while in reality the entire competition is simply not ready.

Do you have a timeline regarding the completion of the competition's starting kit? The blogposts make it seem like it can be worked on already.

For a complete starting kit, I would expect clarifications on the following:

• what is the input/output - do the datasets necessarily constitute as classification-only? Are CIFAR images the benchmark?
• what is implemented that constitutes as an "unlearning" algorithm? Is it the function call unlearning(net, retain, forget, validation), do the networks need to be pre-trained? Is there access to the original training methods?
• what is being evaluated - imo privacy is to be evaluated on the models trained without forget_set, yet there is no code path for that in the notebook. Related to #9 https://github.com/unlearning-challenge/starting-kit/issues/9 .

Please let me know if you need help.

Thanks, Mimee

— Reply to this email directly, view it on GitHub https://github.com/unlearning-challenge/starting-kit/issues/6#issuecomment-1622385644, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3R53NPLRQJRJBZSS2YBI3XOW7SVANCNFSM6AAAAAAZ4W3244 . You are receiving this because you commented.Message ID: @.***>

[eleniTriantafillou]

Hi,

As a follow-up to this, we have updated our notebook to include a comparison against the retrain-from-scratch model (see the last section in the notebook). This can be seen as obtaining a reference point for how well the "ideal" algorithm would perform w.r.t the simple MIA that we're using for evaluation in the notebook.

(It should be emphasized again, though, that our competition will use a different dataset and different metric to evaluate the success of unlearning compared to the ones we use in the starting kit notebook; see our website for updates as they become available.)

Thank you again for your comments.

-Eleni