download pre-train Resnet on CIFAR 10

[FightingFighting]

Hi, Thank you for a great job. could you provide your pre-trained mode? (Resnet on CIFAR 10), that could help for a fair comparison.

[a-F1]

Thank you for your interest in our work. In fact, we did not use a pre-trained model in our work. For a fair comparison, you need to use main_train.py to train an origin model first, and then unlearn it on this basis. This only takes a few minutes. If you have any other questions, please feel free to let us know.

[FightingFighting]

Thank u for the quick reply. so u use main_train.py to pre-train the model, and then use generate_mask.py to generate the Saliency Map, and then use Saliency Map to unlearn, right? Is there any argument that I have to input for main_train.py?

[a-F1]

Yes, your understanding is completely correct! The training script is as follows: python main_train.py --arch resnet18 --dataset cifar10 --lr 0.1 --epochs 182 I will further update our readme to enable more people to understand and use our repo more quickly.

[FightingFighting]

Thank u very much. I have another question. do you use the pre-trained mode (trained by yourself) to generate Saliency Map? if so, there is a mistake in generate_mask.py. it seems like that generate_mask.py dont use the pre-trained model.

[a-F1]

Thank you for your interest in our work! When using generate_mask.py, you need to enter the path of the original model using the --mask parameter.

[FightingFighting]

yes, I use this: python generate_mask.py --save ${saliency_map_path} --mask ${origin_model_path} --num_indexes_to_replace 4500 --unlearn_epochs 1. However, it does not use the pre-trained model.

[a-F1]

That's strange. Could you send me your specific command?

[FightingFighting]

i think the problem is here in generate_mask.py:

    if args.resume:
        checkpoint = unlearn.load_unlearn_checkpoint(model, device, args)

    if args.resume and checkpoint is not None:
        model, evaluation_result = checkpoint
    else:
        checkpoint = torch.load(args.mask, map_location=device)
        if "state_dict" in checkpoint.keys():
            checkpoint = checkpoint["state_dict"]

        if args.unlearn != "retrain":
            model.load_state_dict(checkpoint, strict=False)

        save_gradient_ratio(unlearn_data_loaders, model, criterion, args)

if I use "python generate_mask.py --save ${saliency_map_path} --mask ${origin_model_path} --num_indexes_to_replace 4500 --unlearn_epochs 1", it will not run: model.load_state_dict(checkpoint, strict=False).

[a-F1]

If you do not specify the resume parameter in the command, I think the code that loads the origin model will be run.

[FightingFighting]

what the --unlearn should be? when i run generate_mask.py?

[FightingFighting]

@a-F1

[a-F1]

Sorry for the slow response. When generating the mask, we need to use GA as the unlearn method.

[FightingFighting]

Thank you very much!

[FightingFighting]

Hi after generating map, I got 10 maps (with 0.1, 0.2, 0.3 ... 1.0). which one I should use?

[a-F1]

Hi! Thank you for your interest in our work. Could you please provide more details about the settings, such as the model, the dataset, the forgetting data, etc. The parameter selection may vary under different settings.

[FightingFighting]

Hi， now I am running the code according to your instruction in README. it seems like running with RESNET-18, cifar-10 and 10% forget. everything is the same with your instruction.

[a-F1]

Thank you very much for sharing your settings with me. For the experiments of ResNet18 on Cifar10 with 10% forget, I would recommend using a map of 0.1 paired with a learning rate of 6e-3 for unlearning. Additionally, for other unlearning scenarios, I strongly suggest conducting hyperparameter searches for the sparsity ratio and learning rate. This approach should yield better performance.

[FightingFighting]

Hi, Thank you very much. But could you explain why I used your code and followed your instruction and finally got a very bad result which is rather different from the results reported in your paper?

accuracy : {'retain': 85.44, 'forget': 19.56, 'test': 81.12}
----------------------------------------
SVC_MIA_forget_efficacy : {'correctness': 19.56, 'confidence': 27.04, 'entropy': 56.87, 'm_entropy': 41.02, 'prob': 51.0}
----------------------------------------
SVC_MIA_training_privacy : {'correctness': 52.17, 'confidence': 52.52, 'entropy': 51.58, 'm_entropy': 49.88, 'prob': 50.69}
----------------------------------------

[a-F1]

Thank you for sharing the latest experimental results with me. The results appear to be somewhat over-forgetting, meaning that both TA(Test Accuracy) and RA(Remaining Accuracy) are significantly lower than the Retrained model. I suggest that you consider the following aspects and double-check the details when replicating the paper:

1. What is the accuracy of the origin model? You need to ensure that the accuracy of the origin model is above 94% before proceeding with the unlearning.
2. If possible, could you provide the specific commands you used during the unlearning process?
3. Due to this over-forgetting phenomenon, I recommend reducing the learning rate appropriately when conducting further experiments.

If you encounter any other issues, please feel free to contact us.

[FightingFighting]

Hi,

This is my result for the original model:

"Best epoch": 166, 
"Best Val acc": 94.61999985351562, 
"Train acc": 99.99333333333334, 
"Test acc": 94.69}

For the command: I just use exactly the same setting in your instruction in the readme. generate map:

python generate_mask.py --arch resnet18 --dataset cifar10 --save_dir ${saliency_map_path}  --mask ${origin_model_path} --unlearn GA --num_indexes_to_replace 4500  --unlearn_epochs 1 

unlearn:

python -u main_random.py  --arch resnet18 --dataset cifar10 --unlearn RL  --unlearn_epochs 10 --unlearn_lr 6e-3 --num_indexes_to_replace 4500 --mask ${origin_model_path} --save_dir ${save_dir} --path ${saliency_map_path} 

Then I get the result:

accuracy : {'retain': 99.86, 'forget': 3.09, 'test': 94.1}
----------------------------------------
SVC_MIA_forget_efficacy : {'correctness': 3.09, 'confidence': 16.36, 'entropy': 23.4, 'm_entropy': 34.4, 'prob': 18.04}
----------------------------------------
SVC_MIA_training_privacy : {'correctness': 52.62, 'confidence': 55.78, 'entropy': 55.96, 'm_entropy': 55.66, 'prob': 55.04}
----------------------------------------

Could u provide the hyperparameters u used in your paper?

AND the MIA you reported is SVC_MIA_forget_efficacy ['confidence'], right?

[a-F1]

Yeah! MIA corresponds to SVC_MIA_forget_efficacy ['confidence']. For a detailed clarification on MIA, please refer to Appendix C.3 available at the following link: https://arxiv.org/abs/2304.04934.

I've noticed that the new results are quite close to those presented in our paper. I suggest making a slight adjustment to the unlearn_lr to achieve better performance. The default unlearn_lr was set to 0.1 in the previous README, which might cause some confusion for those trying to replicate the results. I will update the README as soon as possible.

If you encounter any further issues, please don't hesitate to contact us.

[FightingFighting]

Thank you very much!