search

uber / causalml

Uplift modeling and causal inference with machine learning algorithms
Other
4.87k stars 756 forks source link

Qini score > 1 on IT dataset #704

Closed NHUV closed 7 months ago

NHUV commented 7 months ago

Hello,

I computed the qini score on my IT, OOT, OOS and OOT/OOS dataset. I trained a BaseXRegressor using a RandomForestRegressor. When calling qini_score(df, normalize=True) I obtain a qini score of 1.2 on the IT dataset. The qini scores on the OOS, OOT and OOT/OOS are smaller than 1. The df contains 3 columns: 'y', 'w', 'learner'. I understood that, when normalized, the qini score can't be larger than 1. Any suggestions on what might go wrong?

Thanks!

jeongyoonlee commented 7 months ago

Thanks for your question. The normalized Qini and AUUC scores can be greater than 1 when the treatment effect of a subpopulation is negative. The discussions on the previous question in #407 and #524 might be helpful.

NHUV commented 7 months ago

Thanks a lot @jeongyoonlee. I missed those two previous questions.