Add support for diabetes dataset

Trusted-AI / AIF360

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

https://aif360.res.ibm.com/

Apache License 2.0

2.46k stars 840 forks source link

Add support for diabetes dataset #348

Open anupamamurthi opened 2 years ago

anupamamurthi commented 2 years ago

dataset location: https://archive.ics.uci.edu/ml/datasets/diabetes+130-us+hospitals+for+years+1999-2008

Potential Tasks:

[ ] Ensure the license permits open source us
[ ] Verify that this dataset is appropriate for fairness tasks and subset it accordingly (removing un-necessary columns etc.)
[ ] Ensure we have instance level records with protected attributes and outcomes
[ ] First create sklearn-compatible dataset (dataframe) and an appropriate "classic" dataset (second priority)
[ ] Create a simple notebook where the dataset is consumed and simple fairness measures and computed at least.
[ ] DO NOT download and incorporate the data, rather include a function that will do this since data is not hosted in AIF360.

hakimamarullah commented 2 years ago

Hi, @anupamamurthi. I am new to contributing to open source. I would like to work on this one.

hakimamarullah commented 2 years ago

Hi, @anupamamurthi. I've got some troubles implementing this feature. standardize_dataset returned 0 row but kept the columns. here is the test result:

expected dataset.X.shape to be (101766, 48) actual (0, 48)

I thinks it's because of dropna set to True, but when I changed dropna to False, it returned (101766, 13). How do I fix this ?

hakimamarullah commented 1 year ago

Potential task check:

[x] Ensure the license permits open source us
[x] First create sklearn-compatible dataset (dataframe) and an appropriate "classic" dataset (second priority)
[x] DO NOT download and incorporate the data, rather include a function that will do this since data is not hosted in AIF360.
[ ] Verify that this dataset is appropriate for fairness tasks and subset it accordingly (removing un-necessary columns etc.)
[ ] Ensure we have instance level records with protected attributes and outcomes
[ ] Create a simple notebook where the dataset is consumed and simple fairness measures and computed at least.