Using multiclass VA without scikit-learn estimator?

[karllandheer]

Hello, I would like to use multi-class VA without a scikit-learn estimator. In short, I have two lists of calibration output softmax values from a neural net as well as the ground truth labels (N classes) -- one for calibration, one for testing. I have previously used class VennAbers for binary classification with similar data (obviously just 2 classes), however the functionality of VennAbersMultiClass() is different -- it requires a scikit-learn estimator, not just calibration / test data. Is there a way to use your VA implementation to achieve what I want? To convert softmax scores to calibrated multi-class probabilities? Or am I missing something?

Thanks again for all your help!

[ip200]

Hi, thank you for this request. At the moment The multi-class Venn-ABERS calculation relies on the use of one-vs-one classification. This is achieved by passing the underlying scikit-learn classifier into the OneVsOnceClassifier wrapper which converts them into an equivalent set of binary-class classifiers per pair of classes. While this functionality is readily available in scikit-learn, as far as I know the equivalent functionality does not currently exist in Tensorflow or PyTorch. I will investigate some alternative options which could be more general and which could enable you to use the standard softmax outputs to get multi-class Venn Abers probabilities without relying on scikit-learn OneVsOneClassifier functionality. I'll keep you posted, thanks

[ip200]

Hi I am copying you in the response to the same issue raised by tuvelofstrom. I have created a new branch with added functionality which can be found here:

https://github.com/ip200/venn-abers/tree/multiclass_general

Please feel free to try the code above when you have a sec, it allows for pre-fitted multiclass probabilities to be calibrated using the Venn-ABERS method, without the need to supply an underlying scikit learn classifier.

Please also note the following:

• for a given dataset and classifier, this method will give slightly different results to the equivalent IVAP procedure, as the latter fits a new instance for the classifier using scikit learn OnevsOne to the proper training set, whereas this method approximates one-vs-one probabilities from the pre-fitted training and calibration sets

• for very large datasets you can also supply the precision parameter to the VennAbersCalibrator class here too, which speeds up the computation significantly, in addition, you cal also specify p0_p1_output = True in predict_proba which will return a set of p0 and p1 values for all binary pair combinations for the multiclass dataset

Once you had a chance to review, I can easily merge this into the main branch. Thanks

[ip200]

Hi please see version venn-abers 1.4.2 on PyPy pip install venn-abers==1.4.2 . Example of the new method can be found in https://github.com/ip200/venn-abers/blob/main/examples/multiclass_classification.ipynb cell 20. Hope this helps!