Model Class Prediction Probability Thresholds

[AndrewKostandy]

Hi,

Thank you for this nice package!

I wanted to ask if there was a way to modify the probability threshold of class prediction for each model?

So in order to calculate metrics such as TP, FP, TN, and FN, you must be using some prediction probability cutoff value (probably 0.5) to determine a positive vs a negative class prediction.

But there may be a better probability threshold that we've identified for each model and we would like to assess the fairness for each model based on its own optimum threshold. Perhaps something like the argument I added below. Is that possible?

Thanks!

fobject <- fairness_check(
                  rf_mod_explainer, xgb_mod_explainer,
                  protected = attrition_train$gender,
                  privileged = "Male",
                  model_prob_threshold = list(rf_mod_explainer = 0.25, xgb_mod_explainer = 0.43) # like this
)

print(fobject)
plot(fobject)

[jakwisn]

Hi, yes, there is such an option, but with a little different syntax.
First you need to create first fairness object, then second one, and you then need to merge those together.

fobject1 <- fairness_check(
                  rf_mod_explainer,
                  protected = attrition_train$gender,
                  privileged = "Male",
                  cutoff = 0.25 # like this
)
fobject2 <- fairness_check(
                  xgb_mod_explainer,
                  protected = attrition_train$gender,
                  privileged = "Male",
                  cutoff = 0.43 # like this
)

fobject <- fairness_check(fobject1, fobject2)

print(fobject)
plot(object)

It needs to have such syntax because when you would provide cutoff as list, you could set cutoffs for each subgroup. Setting numerical value just sets all cutoffs for subgroups to this value. To see it in work you can check out second vignette - where gmb_cutoff fobject is added.

Hope this helped

[AndrewKostandy]

Thank you Jakub for your clear and quick response!