Question about the toolbox

[TheMrguiller]

Hi @Jianguo99,

I am new to conformal prediction, and I have a multitask, multi-output model that performs both classification and regression for a specific problem. Is it possible to use this kind of model with your tool? I find the applicability of performing two different conformal prediction tasks simultaneously really interesting.

Also, can it be use for a binary classification that was obtain via sigmoid function or it should be always a softmax class probabilities?

[Jianguo99]

Dear TheMrguiller,

Thank you for bringing this issue to our attention. We appreciate your feedback and suggestions.

TorchCP doesn't support the models that perform both classification and regression. To address this, we recommend applying TorchCP separately to the classification and regression components of such models.

Regarding binary classification problems, TorchCP was initially designed for multi-class classification. To this end, we will be updating the framework in the coming days to incorporate the sigmoid function for binary classification tasks. This will provide users with more flexibility and better support for a wider range of classification problems.

We are grateful for your advice. Please let us know if you have any more questions.

Best regards, authors

[TheMrguiller]

It's great to hear that you will implement my suggestion; thank you so much @Jianguo99 . Regarding the multitask models, are there any examples for this case, or should I modify the base class so that TorchCP can determine which logits to use? Also, do you have any recommended readings on Conformal Prediction that you personally endorse?

[Jianguo99]

Dear TheMrguiller,

Unfortunately, I do not have any specific examples of using TorchCP for such scenarios. However, I can provide some general guidance on how to approach this.

As you mentioned, one way to apply TorchCP to a multitask problem is to separate the tasks and use TorchCP independently for each task. This approach does not require rewriting the base class, as you can simply choose the appropriate logits for the task you want to address. For example, you can use the following code (you can find the full code in TorchCP/tests/test_classification.py ):

predictor = class_predictor(score)
predictor.calculate_threshold(cal_logits, cal_labels, alpha)
print(f"Experiment--Data : ImageNet, Model : {model_name}, Score : {score.__class__.__name__}, Predictor : {predictor.__class__.__name__}, Alpha : {alpha}")
prediction_sets = predictor.predict_with_logits(test_logits)

Furthermore, there are numerous resources available for learning about Conformal Prediction (CP).  Firstly, If you want to get to know CP quickly, "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" is a very good choice. After that, you can read the classical textbook, named "Algorithmic Learning in a Random World", or the lastest papers.

Please let me know if you have any further questions or if there is anything else I can assist you with.

Best regards, authors

[TheMrguiller]

Dear @Jianguo99,

I have observed that for all conformal methods for classification they assume that they are working with hard labels. Is there any implementation that seeks to work with soft labels?

Best regards

[Jianguo99]

Dear TheMrguiller,

You might consider utilizing the risk-control framework [1] to achieve your targets.

Please let me know if you have any further questions or if there is anything else I can assist you with.

Best regards, authors

[1] https://arxiv.org/pdf/2208.02814