fed-ul: results of running the example federated unlearning script

mo-shahab commented 9 months ago

Screenshot (149)

veerasagar commented 9 months ago

Screenshot (160)

veerasagar commented 9 months ago

============================================================ Step1. Federated Learning Settings We use dataset: mnist for our Federated Unlearning experiment.

============================================================ Step2. Client data loaded, testing data loaded!!! Initial Model loaded!!! Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to C:\dev\vershachi-unlearning\datasets\mnist\MN IST\raw\train-images-idx3-ubyte.gz 100%|██████████████████████████████████████████████████████| 9912422/9912422 [00:03<00:00, 2977539.26it/s] Extracting C:\dev\vershachi-unlearning\datasets\mnist\MNIST\raw\train-images-idx3-ubyte.gz to C:\dev\vershachi-unlearnin g\datasets\mnist\MNIST\raw

Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to C:\dev\vershachi-unlearning\datasets\mnist\MN IST\raw\train-labels-idx1-ubyte.gz 100%|███████████████████████████████████████████████████████████████████████| 28881/28881 [00:00<?, ?it/s] Extracting C:\dev\vershachi-unlearning\datasets\mnist\MNIST\raw\train-labels-idx1-ubyte.gz to C:\dev\vershachi-unlearnin g\datasets\mnist\MNIST\raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to C:\dev\vershachi-unlearning\datasets\mnist\MNI ST\raw\t10k-images-idx3-ubyte.gz 100%|██████████████████████████████████████████████████████| 1648877/1648877 [00:00<00:00, 3324126.34it/s] Extracting C:\dev\vershachi-unlearning\datasets\mnist\MNIST\raw\t10k-images-idx3-ubyte.gz to C:\dev\vershachi-unlearning \datasets\mnist\MNIST\raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to C:\dev\vershachi-unlearning\datasets\mnist\MNI ST\raw\t10k-labels-idx1-ubyte.gz 100%|█████████████████████████████████████████████████████████████████████████| 4542/4542 [00:00<?, ?it/s] Extracting C:\dev\vershachi-unlearning\datasets\mnist\MNIST\raw\t10k-labels-idx1-ubyte.gz to C:\dev\vershachi-unlearning \datasets\mnist\MNIST\raw

============================================================ Step3. Fedearated Learning and Unlearning Training...

Federated Learning Start

Global Federated Learning epoch = 0 Global Federated Learning epoch = 1 Global Federated Learning epoch = 2 Global Federated Learning epoch = 3 Global Federated Learning epoch = 4 Global Federated Learning epoch = 5 Global Federated Learning epoch = 6 Global Federated Learning epoch = 7 Global Federated Learning epoch = 8 Global Federated Learning epoch = 9 Global Federated Learning epoch = 10 Global Federated Learning epoch = 11 Global Federated Learning epoch = 12 Global Federated Learning epoch = 13 Global Federated Learning epoch = 14 Global Federated Learning epoch = 15 Global Federated Learning epoch = 16 Global Federated Learning epoch = 17 Global Federated Learning epoch = 18 Global Federated Learning epoch = 19

Federated Learning End

Federated Unlearning Start

Federated Unlearning Global Epoch = 0 Local Calibration Training epoch = 5 Federated Unlearning Global Epoch = 1 Federated Unlearning Global Epoch = 2 Federated Unlearning Global Epoch = 3 Federated Unlearning Global Epoch = 4 Federated Unlearning Global Epoch = 5 Federated Unlearning Global Epoch = 6 Federated Unlearning Global Epoch = 7 Federated Unlearning Global Epoch = 8 Federated Unlearning Global Epoch = 9 Federated Unlearning Global Epoch = 10 Federated Unlearning Global Epoch = 11 Federated Unlearning Global Epoch = 12 Federated Unlearning Global Epoch = 13 Federated Unlearning Global Epoch = 14 Federated Unlearning Global Epoch = 15 Federated Unlearning Global Epoch = 16 Federated Unlearning Global Epoch = 17 Federated Unlearning Global Epoch = 18 Federated Unlearning Global Epoch = 19

Federated Unlearning End

Federated Unlearning without Calibration Start

Federated Unlearning without Clibration Global Epoch = 0 Federated Unlearning Global Epoch = 1 Federated Unlearning Global Epoch = 2 Federated Unlearning Global Epoch = 3 Federated Unlearning Global Epoch = 4 Federated Unlearning Global Epoch = 5 Federated Unlearning Global Epoch = 6 Federated Unlearning Global Epoch = 7 Federated Unlearning Global Epoch = 8 Federated Unlearning Global Epoch = 9 Federated Unlearning Global Epoch = 10 Federated Unlearning Global Epoch = 11 Federated Unlearning Global Epoch = 12 Federated Unlearning Global Epoch = 13 Federated Unlearning Global Epoch = 14 Federated Unlearning Global Epoch = 15 Federated Unlearning Global Epoch = 16 Federated Unlearning Global Epoch = 17 Federated Unlearning Global Epoch = 18 Federated Unlearning Global Epoch = 19

Federated Unlearning without Calibration End

Learning time consuming = 1472.0603301525116 secods Unlearning time consuming = 1138.9415872097015 secods Unlearning no Cali time consuming = 0.8456015586853027 secods Traceback (most recent call last): File "C:\Users\dell\vershachi-unlearning\examples\example_fed_unlearn.py", line 160, in Federated_Unlearning() File "C:\Users\dell\vershachi-unlearning\examples\example_fed_unlearn.py", line 114, in Federated_Unlearning old_GMs, unlearn_GMs, uncali_unlearn_GMs = federated_learning_unlearning( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: too many values to unpack (expected 3)

veerasagar commented 9 months ago

Screenshot (161)

============================================================
Step1. Federated Learning Settings
 We use dataset: mnist for our Federated Unlearning experiment.

============================================================
Step2. Client data loaded, testing data loaded!!!
       Initial Model loaded!!!
============================================================
Step3. Fedearated Learning and Unlearning Training...
#####  Federated Learning Start#####
Global Federated Learning epoch = 0
Global Federated Learning epoch = 1
Global Federated Learning epoch = 2
#####  Federated Learning End#####

#####  Federated Unlearning Start  #####
Federated Unlearning Global Epoch  = 0
Local Calibration Training epoch = 2
Federated Unlearning Global Epoch  = 1
Federated Unlearning Global Epoch  = 2
#####  Federated Unlearning End  #####

#####  Federated Unlearning without Calibration Start  #####
Federated Unlearning without Clibration Global Epoch  = 0
Federated Unlearning Global Epoch  = 1
Federated Unlearning Global Epoch  = 2
#####  Federated Unlearning without Calibration End  #####
 Learning time consuming = 44.63474225997925 secods
 Unlearning time consuming = 17.267284870147705 secods
 Unlearning no Cali time consuming = 0.04603409767150879 secods
============================================================
Step4. Membership Inference Attack aganist GM...

Epoch  = -1
Attacking against FL Standard
MIA Attacker precision = 0.9586
MIA Attacker recall = 0.9267
Attacking against FL Unlearn
MIA Attacker precision = 0.5166
MIA Attacker recall = 0.1817

fixed

mo-shahab / vershachi-unlearning