search

scikit-learn-contrib / MAPIE

A scikit-learn-compatible module to estimate prediction intervals and control risks based on conformal predictions.
https://mapie.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
1.24k stars 101 forks source link

Quantile crossing warning #248

Closed nmdwolf closed 1 year ago

nmdwolf commented 1 year ago

Describe the bug

When trying to use the MapieQuantileRegressor class I get the following UserWarnings:

UserWarning: WARNING: The initial prediction values from the quantile method present issues as the upper quantile values might be higher than the lower quantile values.

UserWarning: WARNING: Following the additional value added to have conformal predictions, the upper and lower bound present issues as one might be higher or lower than the other.

The upper quantile better be greater than (or equal to) the lower quantile, so the former warning definitely does not make any sense and the latter one is rather vague. What is meant by the additional value?

To Reproduce Steps to reproduce the behavior:

Expected behavior

I know that quantile regressors can suffer from the quantile crossing problem (without any modifications). However, with the current warning it is hard to see whether this is the problem that occurs or something else is going on.

Desktop

Screenshot image

gmartinonQM commented 1 year ago

Yes, I agree, it is difficult to know if the quantiles cross each other or if the prediction cross the quantiles.

Furthermore, the message is written as a tautology : "the upper quantile value might be higher than the lower quantile value". This is the opposite that should trigger the warning.

Finally, it appears that the warning comes even when there is no crossing at all, as can be simply verified on the testing set : 

(y_pis[:, 0, 0] > y_pis[:, 1, 0]).sum()

I think that this is due to a wrong unfolding of the arguments at this line, the y_pred should be in last position according to the code in predict

https://github.com/scikit-learn-contrib/MAPIE/blob/e476f7fcae28563ea585a4d093dbeae766af017f/mapie/utils.py#L459

nmdwolf commented 1 year ago

I agree. So there are two problems:

  1. On line 459 the arguments are unfolded in the wrong way.
  2. The error messages should read "lower quantile values might be higher than the upper quantile values" instead of "upper quantile values might be higher than the lower quantile values"