search

ip200 / venn-abers

Python implementation of binary and multi-class Venn-ABERS calibration
MIT License
104 stars 9 forks source link

Venn-Abers under ambiguous ground truth? #20

Open karllandheer opened 1 month ago

karllandheer commented 1 month ago

Hello, recently there has been an effort to extend conformal predictors in the case of ambiguous ground truth, namely where labels are sampled from the ground truth distribution, but are not necessarily ground truth themselves: https://arxiv.org/abs/2307.09302. Is there any intention of VA being able to accommodate this? I recognize this isn't necessarily within the scope of the project, but it would be extremely useful as its not uncommon to not have access to ground truth in many tasks. Thanks again for the great package

ip200 commented 1 month ago

Hi thank you very much for flagging this very interesting paper. I believe VA could be modified to account for the scenarios in which labels are distributed according to multiple expert opinions rather representing ground truths. I'll have a think computationally how to best accommodate this - will work on it and drop you a line when I have a branch ready.