search

pymetrics / audit-ai

detect demographic differences in the output of machine learning models or other assessments
MIT License
312 stars 46 forks source link

Issue with bias_test_check in recent version #15

Closed elinorbron closed 5 years ago

elinorbron commented 5 years ago

I am using the current master version.

When the result array passed to bias_test_check contains only two types of grades, for example:(0,0,1), and a value of the parameter test_thresh is not explicitly passed, the function behavior leads to an error.

The default behavior is to set the median value as a threshold - in the case of the input example I gave, that value would be 0.

In such case all students, for example, pass, and the contingency metrics created will be one dimensional. The (only) threshold is then skipped and not returned, and a key error is raised on line 79 in auditai.misc .

danieldiamond commented 5 years ago

Thanks for raising the issue @elinorbron. I've created a PR, modifying the series of statistical checks to handle situations where the threshold is skipped. Feel free to review.