search

gchers / fbleau

A tool for estimating a system's information leakage via Machine Learning
MIT License
10 stars 2 forks source link

fbleau leakage between learn and eval sets (python run_fbleau) #13

Closed EliottZemour closed 2 years ago

EliottZemour commented 2 years ago

I'm using fbleau on log-losses features for membership inference attacks (mia), and I noticed that fbleau consistently beats random guessing on control models: constant (losses = np.ones) and random.

Here's the gist to reproduce the results with python run_fbleau: I obtain attackers accuracy > 0.5 on most of the control models: https://gist.github.com/EliottZemour/2ed30b875be5cfe427bc44b55ec74f01

gchers commented 2 years ago

Thanks a lot for opening this issue, Elliott! I'll look into it -- provisionally marking as #bug.

gchers commented 2 years ago

Hi Eliott! As I mentioned to Bogdan in private msg, the issue seems that you're using min-estimate as opposed to last-estimate. The difference is the following. fbleau estimates the risk (on a fixed test set) for various sizes of the training set. Clearly, this estimate will have some variability, even in the "random data" case that you consider; (imagine outputting predictions uniformly at random: you wouldn't get exactly 50% error). By selecting min-estimate, you're taking the best error observed for some training set size. In fbleau, I recommend this for evaluating the strength of a defence against an attacker: we'd rather overestimate their power than underestimate it. However, what you want is to assert that fbleau is not biased; in this case, you need to use last-estimate, which represents the estimate obtained by training on the entire training set.

I will make sure that the documentation reflects this. Thanks a lot for rising this issue, and let me know if you encounter any other problem.