search

scikit-learn-contrib / MAPIE

A scikit-learn-compatible module to estimate prediction intervals and control risks based on conformal predictions.
https://mapie.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
1.3k stars 110 forks source link

Temperature scaling before conformal prediction leads to unnecessary increase in the width of the prediction sets #483

Closed valeman closed 1 month ago

valeman commented 4 months ago

A new paper shows that temperature scaling before conformal prediction in several popular models implemented in MAPIE including RAPS leads to unnecessary increase in the width of predictions sets. It appears RAPS paper got this incorrect, the need for temperature scaling was not investigated or explained in the RAPS paper.

IMG_3625

It might be worthwhile to review this issue to either remove temperature scaling or to give it as an option explaining to the use the effects.

https://arxiv.org/abs/2402.05806

vincentblot28 commented 1 month ago

Hi @valeman, we are actually not performing any platt scaling for the RAPS method. Before implementing this method we did run some tests with and without scaling and it happened to be unnecessary as you mentioned !

Vincent