I previously pushed these notebooks to master but this updates them, and the PR describes them.
This includes two notebooks:
generate_disclosure_risk.ipynb calls synthimpute functions to generate disclosure risk datasets for synpuf7 and synthpop_samp. These are keyed on the synthetic ID, with distances and IDs for nearest records in the train and test sets.
disclosure_risk.ipynb loads those disclosure risk datasets and evaluates the distribution and relationships of train_dist and test_dist for both synpuf7 and synthpop_samp: scatterplots of train_dist vs. test_dist, CDFs, and shares of exact matches to the train and test sets.
My main takeaway is that, at least on this size dataset and with these specifications, synthpop outperforms random forests:
It produces records that are slightly less similar to training records, and slightly closer to test records (see cells 7, 8, and 13). Note that both models overfit to some degree: train_dist is statistically significantly lower than test_dist.
7.7% of synthpop records exactly match a training record, vs. 13.5% of RF records.
Of these exact synthpop matches to training records, 50% also exactly match test records, vs. 25% of RF.
0.9% of synthpop records exactly match a test record without matching a training record, vs. 0.4% of RF records. We want this to be high.
NB: Cells 4-6 of disclosure_risk.ipynb contain synth_id and how close it is to a training record. This is for informational purposes, but could itself be a disclosure risk if we were to publish a synthetic file. Since those IDs and records will differ in the final file, I think it's OK to keep here for now, but a final analysis should omit them.
I've also added to the restricted Drive folder synpuf_disclosure_risk_with_record_exam.ipynb (this link will only be accessible to the team). This notebook shows four example synthetic records compared to their nearest train and test records for both synthpop and RF:
A "typical" record with values of all relevant distance metrics between the 40th and 60th percentile. These didn't tend to be worryingly similar; more variables didn't match exactly than those that did.
Synthetic records which exactly match both a training record and a test record. These were very simple records which look like they could represent many thousands of taxpayers. (Aside: we should remove duplicate records and add weight for faster execution.)
The synthetic record which exactly matches a training record and has the largest distance from a test record. These are potential disclosure risks, and we should monitor them when synthesizing larger datasets. We could also explore mitigating approaches like synthesizing more records and discarding exact matches.
The synthetic record which exactly matches a test record and has the largest distance from a training record (we like these). These were relatively simple records too.
One other note: I split the distances by MARS/DSI/XTOT as anything less was crashing my Python kernel. So this might be a bit more of a rosy picture than we'd get if we did the full distance matrix. I'll be looking into optimizations to make this possible.
I previously pushed these notebooks to master but this updates them, and the PR describes them.
This includes two notebooks:
synpuf7andsynthpop_samp. These are keyed on the synthetic ID, with distances and IDs for nearest records in the train and test sets.train_distandtest_distfor bothsynpuf7andsynthpop_samp: scatterplots oftrain_distvs.test_dist, CDFs, and shares of exact matches to the train and test sets.My main takeaway is that, at least on this size dataset and with these specifications,
synthpopoutperforms random forests:train_distis statistically significantly lower thantest_dist.NB: Cells 4-6 of disclosure_risk.ipynb contain
synth_idand how close it is to a training record. This is for informational purposes, but could itself be a disclosure risk if we were to publish a synthetic file. Since those IDs and records will differ in the final file, I think it's OK to keep here for now, but a final analysis should omit them.I've also added to the restricted Drive folder synpuf_disclosure_risk_with_record_exam.ipynb (this link will only be accessible to the team). This notebook shows four example synthetic records compared to their nearest train and test records for both synthpop and RF:
One other note: I split the distances by MARS/DSI/XTOT as anything less was crashing my Python kernel. So this might be a bit more of a rosy picture than we'd get if we did the full distance matrix. I'll be looking into optimizations to make this possible.