Question regarding THR

[ThomasNorr]

Hello,

thanks for your very useful toolbox. Since you seem to be very experienced with conformal prediction, could you give me an idea about the weakness of the THR score function, when combined with the SplitPredictor?

Across a range of models I tested, it seems to provide a vastly smaller average_size (Easily half of APS) for a given reached coverage_rate. Is it "just", as mentioned in the RAPS Paper (https://arxiv.org/abs/2009.14193), the theoretical guarantee of the coverage rate?

Additionally, I was wondering if the "naive" method of this paper (https://arxiv.org/pdf/2306.09335) is implemented here as well under a different name (Is it Margin?)?

Thanks again for your great work :)

Regards Thomas

[hongxin001]

Dear Thomas，

Thank you for the interest in this toobox. We list our replies for each question below.

• The weakness of THR. Yes, THR can achieve excellent performance in the average size. However, it generally perform bad in conditional coverage, as shown in Table 11 of RAPS [1]. In addition, the introduction of conformal prediction (CP) [2] gives a detailed discussion of THR.
• "Naive" method. We cannot find the "naive" word in the paper [3]. As presented in README.md, we provide the list of implementations and their papers in the table. The method of paper [3] is implemented as "classification.predictors.cluster". The "classification.scores.margin" correspondings to the paper [4].

Please let us know if you have any more questions.

Best regards, authors

[1] Uncertainty Sets for Image Classifiers using Conformal Prediction [2] A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification [3] Class-Conditional Conformal Prediction with Many Classes [4] Bias reduction through conditional conformal prediction

[ThomasNorr]

Thanks for the detailed answers and providing helpful resources. I will include "SSCV" then in my experiments. As a side note, it would be really cool if the metrics contained references as well.

Also, I unfortunately messed up the references. "naive" was in the RAPS paper aswell. I however don't think it is Margin.

Best regards, Thomas

[Jianguo99]

Dear Thomas，

Sorry for the late response. "naive" presented in Raps is not included in TorchCP.

Additionally, we have updated the references for the metrics. We hope this assists you in your research.

Please let us know if you have any more questions.

Best regards, authors