Membership inference attack overall auc is 0.519427

[M-Ghorbel]

I'm training to use your membership inference attack to evalute my own model but i started first by running the code you provided to be familiar with every line.

I got wierd results (i did not change anything in your code)

train_accuracy: 0.996 test_accuracy: 0.5535 (very low accuracy for cifar10)

and

overall auc is 0.519427 (random reslut which is not expected for a model runningh without dp-sgd)

can you please explain what is wrong?

[Koukyosyumei]

Thank you for your report! I'll investigate this issue.

[M-Ghorbel]

thank you, ,please let me know when you solve the problem

Le mer. 1 févr. 2023 à 11:42, Syumei @.***> a écrit :

Thank you for your report! I'll investigate this issue.

— Reply to this email directly, view it on GitHub https://github.com/Koukyosyumei/AIJack/issues/125#issuecomment-1411845635, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWCX6RXUU4MM7DPDLZTM63WVI4YZANCNFSM6AAAAAAUK6KF3Q . You are receiving this because you authored the thread.Message ID: @.***>

[Koukyosyumei]

@M-Ghorbel

I have almost finished the investigation. The test accuracy on CIFAR-10 seems low, but the original paper also reports similar results (around 0.6), indicating that the model is heavily overfitted on their training sets. Note that both our implementation nor the original work didn't use sophisticated SOTA models.

As for the second question, the implementation contains some bugs, like the position of positive estimations within the returned values from sklearn API. I re-implemented this attack and made it compatible with sklearn and PyTorch. Also, I updated the tutorial, which uses MNIST instead of CIFAR-10 since MNIST requires less computational power. You can run this notebook in google colab.

[M-Ghorbel]

Thank you, i will try it.

Le dim. 5 févr. 2023 à 10:02, Syumei @.***> a écrit :

@M-Ghorbel https://github.com/M-Ghorbel

I have almost finished the investigation. The test accuracy on CIFAR-10 seems low, but the original paper also reports similar results (around 0.6), indicating that the model is heavily overfitted on their training sets. Note that both our implementation nor the original work didn't use sophisticated SOTA models.

As for the second question, the implementation contains some bugs, like the position of positive estimations within the returned values from sklearn API. I re-implemented this attack and made it compatible with sklearn and PyTorch. Also, I updated the tutorial https://koukyosyumei.github.io/AIJack/notebooks/aijack_membership_inference.html, which uses MNIST instead of CIFAR-10 since MNIST requires less computational power. You can run this notebook in google colab.

— Reply to this email directly, view it on GitHub https://github.com/Koukyosyumei/AIJack/issues/125#issuecomment-1417144002, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWCX6RET6HPVYEURZYPFN3WV5UBFANCNFSM6AAAAAAUK6KF3Q . You are receiving this because you were mentioned.Message ID: @.***>

[OctopusJIA]

I am also trying the membership inference and flowing the tutorial on 4.1 you proposed, but I didn't find the mnist dataset used in your code.

df = pd.read_csv("/content/sample_data/mnist_train_small.csv", header=None)

Cloud you please tell me where I can get this dataset?

[OctopusJIA]

I used the mnist dataset downloaded from kaggle in csv format to run successfully.