search

Trusted-AI / AIF360

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.
https://aif360.res.ibm.com/
Apache License 2.0
2.41k stars 825 forks source link

make_scorer functionality with equal_opportunity_difference? #300

Open tashahughes opened 2 years ago

tashahughes commented 2 years ago

Hi,

Not sure if this is an issue or more of a request, I can see something similar was suggested a while back: https://github.com/Trusted-AI/AIF360/pull/174

I am trying to cross validate using equal_opportunity_difference, I have managed to train my model and get an fairness score overall, yet I am a bit confused as to how to do this inside sklearn's cross_val_score method. I have tried to make a 'make_scorer' function using sklearn's recommendations, however I get an error: 'dataset should be BinaryLabelDataset or MultiClassLabelDataset'. As far as I am aware, the dataset is a BinaryLabelDataset, otherwise it would not have worked earlier (without cross validation.) Has anyone been able to cross validate like this or has this not been implemented yet?

nrkarthikeyan commented 2 years ago

What version of the metric are you using? There are two versions - one is sklearn compatible and one is not. https://github.com/Trusted-AI/AIF360/blob/faa75ee0cfffb57ecb921b8ea36970e0bda669f5/aif360/sklearn/metrics/metrics.py#L352

https://github.com/Trusted-AI/AIF360/blob/faa75ee0cfffb57ecb921b8ea36970e0bda669f5/aif360/metrics/classification_metric.py#L838