search

awslabs / fortuna

A Library for Uncertainty Quantification.
https://aws-fortuna.readthedocs.io/en/latest/
Apache License 2.0
892 stars 46 forks source link

feat: mondrian conformal prediction #168

Open mcapuccini opened 11 months ago

mcapuccini commented 11 months ago

Feature Request

Describe the Feature Request Looking at the documentation it seems like Fortuna implements Inductive Confomal Prediction. I couldn't understand if you are using a mondrian approac or not, meaning that you are calculating the non-conformity measures for the calibration set (or alphas) for each class separately (and computing conformal p-values for each class).

Describe Preferred Solution If fortuna implements mondrian ICPs it would be good to add it to the documentation otherwise it would be nice to have it done with the mondrian approach for better handling class imbalances.

Related Code n/a

Additional Context n/a

If the feature request is approved, would you be willing to submit a PR? Yes (if time permits, I am not sure if I have the capacity during my working hours)

gianlucadetommaso commented 11 months ago

Hi Marco, thanks a lot for your input! Not, at the moment the Mondrian version is not implemented.

If you did have time to work on this, it would be great! I believe it does not require major changes: you would mostly need to extend this class to allow for a class-dependent threshold.

Let me know if you get the chance to contribute!

mcapuccini commented 11 months ago

Hi Gianluca, thanks for the quick answer. The threshold (sig. level) would remain the same but the p-value would be computed for each class separately according to the calibration examples of the relative class. This means that I would need to look at the true label of the calibration samples to filter out the examples on which the p-value is computed.