Individual notions of fairness: Give similar predictions to similar individuals.
Group notions of fairness: Treat different groups equally.
Subgroup notions of fairness: This notion intends to obtain the best properties of the group and individual notions of fairness, for example, by picking a group fairness constraint and asking whether this constraint holds over a large collection of subgroups.
Equal Opportunity:
The probability of a person in a positive class being assigned to a positive outcome should be equal for both protected and unprotected group members. In other words, the equal opportunity definition states that the protected and unprotected groups should have equal true positive rates [group].
Equalized Odds:
The probability of a person in the positive class being correctly assigned a positive outcome and the probability of a person in a negative class being incorrectly assigned a positive outcome should both be the same for the protected and unprotected group members. In other words, the equalized odds definition states that the protected and unprotected groups should have equal rates for true positives and false positives [group].
Demographic/ Statistical Parity:
The likelihood of an outcome should be the same regardless of whether a person is in the protected group [group].
Test Fairness:
For any predicted probability score S, people in both protected and unprotected groups must have equal probability of belonging in the positive class [group].
Counterfactual Fairness:
This is based on the intuition that a decision is fair towards an individual if it’s the same in both the real world and a counterfactual world where the individual belongs to a different demographic group [individual].
Conditional Statistical Parity:
This states that people in both protected and unprotected groups should have equal probability of being assigned to a positive outcome given a set of legitimate factors [group].
Fairness through Awareness:
An algorithm is fair if it gives similar predictions to similar individuals. That is, any 2 individuals who are similar with respect to a similarity (inverse distance) metric defined for a particular task should receive a similar outcome [individual].
Fairness through Unawareness:
An algorithm is fair as long as any protected attributes A are not explicitly used in the decision-making process [individual].
Treatment Equality:
Treatment equality is achieved when the ratio of false negatives and false positives is the same for both protected group categories [group].
PPV Parity (Positive Predicted Value) equalizes the chance of success given a positive prediction.
FPR-Parity (False Positive Rate) is the exact opposite and ensures that the 2 groups have the same false positive rate.
NPV-Parity (Negative Predicted Value) ensures the ratio of correctly rejecting people out of all the people the model has rejected is the same for each group.
Next Steps
The most research has been done in looking at group notions of fairness so there will be more extensive literature on that, making it easier to find good references. I'd want to have a deep dive on the group definitions of fairness and make that the focus of my thesis. Two in particular that interest me are conditional statistical parity and test fairness.
Following that, Professor Spector pointed me to a group at UMass called EQUATE that does similar fairness research: https://groups.cs.umass.edu/equate/people. Phil Thomas, in particular, has done work on measures of fairness and the ways in which different fairness measures conflict with one another. Neither Professor Spector nor I have identified which of his papers deal most directly with this yet but this is a direction of exploration and simulation that has piqued my interest.
On individual notions of fairness, the definition of counterfactual fairness has sparked some curiosity that I can potentially explore.
Finally, I also began reading on fair representation learning, which is a data debasing technique, but I think this will be more useful slightly later when looking at methods and applications.
@katcorr
Fairness Definitions
Fairness metrics fall under 3 major categories:
Equal Opportunity:
The probability of a person in a positive class being assigned to a positive outcome should be equal for both protected and unprotected group members. In other words, the equal opportunity definition states that the protected and unprotected groups should have equal true positive rates [group].
Equalized Odds:
The probability of a person in the positive class being correctly assigned a positive outcome and the probability of a person in a negative class being incorrectly assigned a positive outcome should both be the same for the protected and unprotected group members. In other words, the equalized odds definition states that the protected and unprotected groups should have equal rates for true positives and false positives [group].
Demographic/ Statistical Parity:
The likelihood of an outcome should be the same regardless of whether a person is in the protected group [group].
Test Fairness:
For any predicted probability score S, people in both protected and unprotected groups must have equal probability of belonging in the positive class [group].
Counterfactual Fairness:
This is based on the intuition that a decision is fair towards an individual if it’s the same in both the real world and a counterfactual world where the individual belongs to a different demographic group [individual].
Conditional Statistical Parity:
This states that people in both protected and unprotected groups should have equal probability of being assigned to a positive outcome given a set of legitimate factors [group].
Fairness through Awareness:
An algorithm is fair if it gives similar predictions to similar individuals. That is, any 2 individuals who are similar with respect to a similarity (inverse distance) metric defined for a particular task should receive a similar outcome [individual].
Fairness through Unawareness:
An algorithm is fair as long as any protected attributes A are not explicitly used in the decision-making process [individual].
Treatment Equality:
Treatment equality is achieved when the ratio of false negatives and false positives is the same for both protected group categories [group].
Comparison Definitions:
Disparate Impact Ratio: P(Y=1 | D=unprivileged) / P(Y=1 | D=privileged) [0.8 to 1.25 fair]
Statistical Parity Difference: P(Y=1 | D=unprivileged) - P(Y=1 | D=privileged) [-0.1 to 0.1 fair]
Equal Opportunity Difference: TPR | D=unprivileged - TPR | D=privileged [-0.1 to 0.1 fair]
Equalized Odds Difference: (FPR | D=unprivileged - FPR | D=privileged) / 2 [-0.1 to 0.1 fair]
Additional Definitions:
PPV Parity (Positive Predicted Value) equalizes the chance of success given a positive prediction.
FPR-Parity (False Positive Rate) is the exact opposite and ensures that the 2 groups have the same false positive rate.
NPV-Parity (Negative Predicted Value) ensures the ratio of correctly rejecting people out of all the people the model has rejected is the same for each group.
Next Steps