Closed sherdencooper closed 3 years ago
Can you give me more details about what your hyperparameters were? In particular, I'm wondering what your size parameter s was.
Thx for replying! I train both a lite VGG referred in your paper and a standard VGG11 with 100 epochs. I sampled randomly 20000 images from CIFAR10 as a distribution. Then 10000 images of the distribution are used to train the model and the rest 10000 images are viewed as non-membership. I evaluated the performance of membership inference attack in the 20000 images. It's weird that the precision is about 75% but the recall rate is only 90%. I'm wondering why the recall rate is much lower, in other words, the inference attack algorithm cannot classify membership data points into membership well.
Our version of the VGG network, as described in our paper, is parametrized by the size parameter s. When we set this value to 128, we got a recall greater than 99%. Which value of s did you use?
In fact, I set s=64 and I also set s=128 to evaluate the attack performances but they didn't show any difference. Can the training strategy lead to this? I mean, such as shuffle the trainset, normalize, random crop or other data augmentation make the model less overfit and thus recall rate drops?
Hi, your research is really interesting! I am reproducing your experiments in CIFAR10. However, while the precision is able to achieve the performance in your paper, the recall rate is much lower than that in your paper (about 90%). This is my step to calculate the recall rate:
However, only about 90% of the data points in S were successfully determined as members. Did i do something wrong? Thx a lot in advance.