search

AI-sandbox / XGMix

13 stars 2 forks source link

What are the implications of the calibrate option? #2

Closed nievergeltlab closed 4 years ago

nievergeltlab commented 4 years ago

The calibrate option does not have much explanation.

In xgmix.py, it says "# calibrates the predictions to be balanced w.r.t. the train1 class distribution"

but it isn't clear what this means. Can you please provide some details about what this option is supposed to do and when it is meant to be used?

Thanks!

RichRast commented 4 years ago

Hi Thanks for reaching out! We just added the description of the calibration parameter in the read me section (also noted below). The parameter is set to True by default and shouldn't need to be changed. Please let us know if you have more questions.

XGmix output probabilities might not reflect the true confidence of the predictions. By setting the calibration parameter to True when training a new model, Isotonic Regression is used to match the predicted probabilities to calibrated probabilities. For example, in a calibrated model, predictions with a probability 80% will be correct 80% of the time.