search

frstyang / fairness-with-overlapping-groups

3 stars 5 forks source link

Support for Equal Opportunity #2

Open niceIrene opened 7 months ago

niceIrene commented 7 months ago

Thank you for sharing the code.

I am exploring using the Equal Opportunity fairness metric for overlapping (gerrymandering) groups. I looked at the gerry_exp.py file and here are my parameter settings:

params_list = [{'B':[10], 'nu':np.logspace(-4,0,20), 'T':[10000], 'lr': [1], 'fairness': ['EO'], 'gfair': ['gerry'], 'lambda_update': ['subgradient']}, {'B':[10], 'nu':np.logspace(-4,0,20), 'T':[10000], 'lr': [1], 'fairness': ['EO'], 'gfair': ['gerry'], 'lambda_update': ['subgradient']}, {'C':[15], 'nu':np.logspace(-3,0,20), 'T':[1500]}]

metrics_list = [ [('accuracy', lambda p,x,xp,y: calc_acc(p,y)), ('gerry_eo', lambda p,x,xp,y: calc_eo_viol(p, xp, y))], [('accuracy', lambda p,x,xp,y: calc_acc(p,y)), ('gerry_eo', lambda p,x,xp,y: calc_eo_viol(p, xp, y))], [('accuracy', lambda p,x,xp,y: calc_acc(p,y)), ('gerry_eo', lambda p,x,xp,y: calc_eo_viol(p, xp, y))] ] I am not sure if any modifications of the "calc_eo_viol" need to be made to handle the gerry group case. It sees something like "calc_gerry_viol2" is needed.

Thank you!

frstyang commented 7 months ago

Only the first two methods in methods = [groupfair.Plugin, groupfair.WERM, gerfair.GerryFair] would support combining gerrymandering groups with equal opportunity, i.e. gerfair.GerryFair and its settings should be removed. However, combining gerrymandering groups with equal opportunity is not recommended in the first place. It will likely be impossible to achieve a nontrivial fair classifier in this case because there are too many groups (exponential in the number of sensitive attributes), and equal opportunity does not do any natural downweighting of small groups like gerrymandering fairness does.

niceIrene commented 7 months ago

Hi Forest,

Thanks for your quick reply. I agree that achieving a fair classifier for all overlapping subgroups is almost impossible.

You mentioned equal opportunity does not have a natural down-weighting of small groups. I am wondering if we can do something like define the violation measure by weighted the groups by their population sizes (as what you did in calc_gerry_viol, e.g. group_size*diff_TP).

I found that GerryFair (Kearns et al.) supports using FP (similar to equal opportunity) as the fairness metric. I am wondering how it differs from the evaluation metrics in this repo. Thanks a lot!