Result of default OptimalBinning is worse compare with the one that has more restrictions

[nic9lif3]

Hi @guillermo-navas-palencia,

Today I found a weird result with OptimalBinning,

If I use this code, the result will give the IV total is 0.001707

bin=optbinning.OptimalBinning(
    min_prebin_size=0.01,
    max_n_prebins=40,
#     monotonic_trend='ascending'
)
bin.fit(data_train['avginstallast24m_3658937A'],data_train['target'])
bin.binning_table.build()

But when I add the monotonic_trend constraint, it will give a better IV 0.005960

bin=optbinning.OptimalBinning(
    min_prebin_size=0.01,
    max_n_prebins=40,
    monotonic_trend='descending'
)
bin.fit(data_train['avginstallast24m_3658937A'],data_train['target'])
bin.binning_table.build()

This data can be found here datatest.csv Could you explain why it happened? Perhaps the fewer contains, the higher the IV? Thanks.

[guillermo-navas-palencia]

Hi @nic9lif3,

The monotonic_trend="auto" is not infallible. In this particular case, the predicted monotonic trend was not optimal. In general, when event rates are close to flat, the algorithm might fail. However, this situation is much more unlikely for features with medium/high IV.