Support Binary classification

[jrash]

It seems only multiclass classification is supported as I see:

assert type_of_target(y) == "multiclass" in classification.py. Seems to me that binary classification could also be supported with the implemented methods.

[vtaquet]

Hi @jrash , thanks for raising this issue. MAPIE works indeed only for multi-class classification since the conformal prediction methods give "prediction sets" only relevant with >= 3 classes. If you are interested in binary classification, sklearn already has some tools for calibration. See for instance CalibratedClassifierCV.

[jrash]

Hi @vtaquet. Thanks for you response. Just wanted to share that conformal prediction sets for binary classification is very popular in drug discovery. Here are some relevant publications:

https://pubs.acs.org/doi/full/10.1021/acs.jcim.7b00159 https://pubs.acs.org/doi/10.1021/ci5001168 https://arxiv.org/abs/1908.03569

What we typically do is actually the Mondrian conformal due to the fact that classes are often highly imbalanced, but there cases when classes are fairly balanced and Mapie may be relevant as is. I looked at references 3-5 in your readme and I see nothing in the methods or theoretical guarantees that precludes binary classification, unless I am missing something.

I think people in our field would benefit from many of the tools you are developing such as the covariate shift work I see going on. So hopefully this library could be useful to our community. But often we are interested in binary classification.

[firmai]

Maybe this package could help as a starting point for integration https://github.com/donlnz/nonconformist

[lcrmorin]

@jrash I am curious about conformal prediction for binary classification. Do you happen to have a demo code for Mondrian Conformal for binary classification ?

[valeman]

It is puzzling to see significant gap in MAPIE when it comes to binary classification.

It appears the focus in MAPIE since inception seems to have been for some reason on multi class methods that are more suitable for computer vision problems rather than tabular data that is way more prevalent in the industry.

Binary classification for tabular data is one of the main (the main?) class of problems in the industry.

Venn-Abers is easy to implement (is available in several python packaged( and is the most performant method for binary classification https://www.diva-portal.org/smash/get/diva2:1603345/FULLTEXT01.pdf

There are several packages that have implemented both classical CP methods and also more recent methods such as VA for binary classification:

1) Newer and faster package - https://github.com/ip200/venn-abers

2) older package that also works well https://github.com/ptocca/VennABERS

3) Nonconformist

4) Crepes

[thibaultcordier]

Hello everyone,

Concerning binary classification, you can see more details about what is possible to do (https://mapie.readthedocs.io/en/stable/theoretical_description_binary_classification.html), in theory, in MAPIE and in other packages like scikit-learn. Thanks @valeman for sharing more references.

Regarding the MAPIE team's roadmap 2024 for binary classification, we'll be proposing the Mondrian Conformal Prediction method (see https://github.com/scikit-learn-contrib/MAPIE/discussions/372).

However, Venn-Abers is not the main focus, but feel free to help us contribute to MAPIE by making a pull request with your proposal. We can discuss together why you think this is needed and if it needs to be integrated into the package.

[firmai]

Venn-Abers would be a great addition would be cool if you can add it Valerie, has it been compared to the Mondarian method for robustness?