ENSTA-U2IS-AI / torch-uncertainty

Open-source framework for uncertainty and deep learning models in PyTorch :seedling:
https://torch-uncertainty.github.io
Apache License 2.0
308 stars 20 forks source link

:sparkles: Implement Adaptive ECE metric #92

Closed qbouniot closed 7 months ago

qbouniot commented 7 months ago

This is a proposition of implementation for the Adaptive ECE metric into torch_uncertainty/metrics/classification/calibration.py. I've also added it by default in the multiclass classification routine. There is also a minor fix when doing temperature scaling with the test set. Since the test datasets are returned as a list, (ID test set + OOD test set), the dataset used will be the first element of the list by default.

TO DO:

Would fix #83.

codecov[bot] commented 7 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.04%. Comparing base (740ae5b) to head (1d0988b). Report is 10 commits behind head on dev.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## dev #92 +/- ## ========================================== + Coverage 99.02% 99.04% +0.01% ========================================== Files 108 109 +1 Lines 5153 5212 +59 Branches 694 702 +8 ========================================== + Hits 5103 5162 +59 Misses 17 17 Partials 33 33 ``` | [Flag](https://app.codecov.io/gh/ENSTA-U2IS-AI/torch-uncertainty/pull/92/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ENSTA-U2IS-AI) | Coverage Δ | | |---|---|---| | [cpu](https://app.codecov.io/gh/ENSTA-U2IS-AI/torch-uncertainty/pull/92/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ENSTA-U2IS-AI) | `99.04% <100.00%> (+0.01%)` | :arrow_up: | | [pytest](https://app.codecov.io/gh/ENSTA-U2IS-AI/torch-uncertainty/pull/92/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ENSTA-U2IS-AI) | `99.04% <100.00%> (+0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ENSTA-U2IS-AI#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

o-laurent commented 7 months ago

Thanks, @qbouniot, for the PR!

@alafage, shouldn't we implement AdaptiveCE as a special case of CE? Just add a new parameter adaptive that would default to False?

Btw, do we add a num_calibration_bins param to the classification routine?

Edit: after discussion with @alafage we settled for yes to both questions.