search

gpleiss / equalized_odds_and_calibration

Code and data for the experiments in "On Fairness and Calibration"
MIT License
49 stars 20 forks source link

FP rate difference diminished, but FN rate difference greatly increased? #2

Open MatthijsdeJ opened 5 years ago

MatthijsdeJ commented 5 years ago

When testing the equalized odds algorithm on my data, I checked the metrics. I found that it only diminishes the difference between FP rates, but it actually greatly increased the difference for FN rates. This does not seem allign with Equalized Odds, AFAIK.

Original group 0 model: Group size: 943.000 Prediction Accuracy: 0.690 FP rate: 0.113 FN rate: 0.196

Original group 1 model: Group size: 641.000 Prediction Accuracy: 0.771 FP rate: 0.062 FN rate: 0.167

Equalized odds group 0 model: Group size: 943.000 Prediction Accuracy: 0.652 FP rate: 0.063 FN rate: 0.285

Equalized odds group 1 model: Group size: 641.000 Prediction Accuracy: 0.761 FP rate: 0.076 FN rate: 0.162

MatthijsdeJ commented 5 years ago

It is worse for the calibrated equalized odds model. Here, both the difference between FP rates and FN rates increase, even when using the 'weighted' option.

Original group 0 model: Group size: 2331.000 Prediction Accuracy: 0.701 FP rate: 0.095 FN rate: 0.203

Original group 1 model: Group size: 1628.000 Prediction Accuracy: 0.740 FP rate: 0.052 FN rate: 0.208

Equalized odds group 0 model: Group size: 2331.000 Prediction Accuracy: 0.701 FP rate: 0.095 FN rate: 0.203

Equalized odds group 1 model: Group size: 1628.000 Prediction Accuracy: 0.741 FP rate: 0.035 FN rate: 0.224