scikit-learn-contrib / MAPIE

A scikit-learn-compatible module to estimate prediction intervals and control risks based on conformal predictions.
https://mapie.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
1.3k stars 111 forks source link

Error when computing multiple prediction intervals in Time Series Tutorial #484

Open PolZT opened 4 months ago

PolZT commented 4 months ago

Hello MAPIE team,

Thanks for this wonderful library. I'm encountering an issue when trying to compute two intervals instead of one in the time series tutorial, but I might be wrong or missing something in my understanding.

Describe the bug

When attempting to compute two intervals instead of one by changing the alpha value from 0.05 to [0.1, 0.05] in the time series tutorial, a ValueError is raised. The error message indicates an issue with evaluating the truth value of an array in the conformity_scores.py file.

To Reproduce

Steps to reproduce the behavior:

  1. Go to the time series tutorial provided in the MAPIE documentation: MAPIE Time Series Tutorial.
  2. Replace the original alpha = 0.05 with alpha = [0.1, 0.05] on line 193 of the tutorial script.
  3. Run the script.
  4. See the error.

Expected behavior

The script should compute and return two prediction intervals without raising any errors.

Screenshots

N/A (the error message is provided below).

Desktop:

Additional context

I tried to find out what is causing it and here are some of my findings:

In conformity_scores.py, the get_quantile function (line 213) receives an inconsistent formatting of alpha_np, sometimes receiving a simple NDArray and sometimes receiving an array of arrays.

Error traceback:

y_pred_enbpi_npfit, y_pis_enbpi_npfit = mapie_enbpi.predict(
  File "C:\Users\pzaho\anaconda3\envs\dynamo\lib\site-packages\mapie\regression\time_series_regression.py", line 459, in predict
    return super().predict(
  File "C:\Users\pzaho\anaconda3\envs\dynamo\lib\site-packages\mapie\regression\regression.py", line 645, in predict
    self.conformity_score_function_.get_bounds(
  File "C:\Users\pzaho\anaconda3\envs\dynamo\lib\site-packages\mapie\conformity_scores\conformity_scores.py", line 454, in get_bounds
    quantile_low = self.get_quantile(
  File "C:\Users\pzaho\anaconda3\envs\dynamo\lib\site-packages\mapie\conformity_scores\conformity_scores.py", line 272, in get_quantile
    quantile = signed * np.column_stack([
  File "C:\Users\pzaho\anaconda3\envs\dynamo\lib\site-packages\mapie\conformity_scores\conformity_scores.py", line 276, in <listcomp>
    ) if not (unbounded and _alpha >= 1) else np.inf * np.ones(n_ref)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()