koaning
commented
4 years ago This definately sounds interesting, and welcome! We've got some metrics from scikit-lego that we will import such as p_percent_score (thought I cannot promise a timeline, considering the pandemic).
It should also be said that at the moment, we're not affiliated with scikit-learn. The end goal is that we can consolidate many seperate fairness packages out there into a single standard for scikit learn and this will be declared as an official "scikit-contrib" package. There's still steps that need to be taken to formalise that and I don't know on what timeframe this might happen (again, pandemic).
nsorros
Hi, I am Nick. Work for WellcomeTrust during the day and volunteer with DataKind UK at night. In both places I am somehow involved with ethics and fairness. Will be happy to contribute in one way or another to this library.
At the Wellcome we have developed a relatively simple fairness report function similar to the classification report of sklearn https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html that computes instead of precision, recall, f1, true positive rate, false positive rate etc per group of interest.
The API atm is
fairness_report(y_true, y_pred, groups, group_names)with groups indicating membership on a group of interest which can also be a matrix if examples can belong to multiple groups in which case group names gives the name of the columns. An additional average parameter can be added to give more control on how these cases are handled which at the moment defaults to micro. Let me know if this is an addition you are interested in.