search

ml-stat-Sustech / TorchCP

A Python toolbox for conformal prediction research on deep learning models, using PyTorch.
GNU Lesser General Public License v3.0
229 stars 32 forks source link

Example update & Missing smooth sorting in ConfTr loss implementation #30

Closed emirceyani closed 1 week ago

emirceyani commented 1 week ago

Hi,

Thanks for having PyTorch implementation of Conformal Training.

According to

  1. David Stutz, Krishnamurthy Dj Dvijotham, Ali Taylan Cemgil, Arnaud Doucet. Learning optimal conformal classifiers. ICLR, 2022.
  2. Training Uncertainty-Aware Classifiers with Conformalized Deep Learning, NeurIPS, 2022

Conformal training includes smooth sorting & ranking. however, in the forward funcrion of ConfTr class, I realized that this part is not differentiable at all. So, this raises a suboptimal implementation.

Also, the MNIST example you provided is not comparable as benchmarking ConfTr paper of stutz et. al., as the MNIST dataset is benchmarked only with a linear model. Could you update your codes to include EMNIST dataset ? (I can try to add EMNIST if I have time also )

Thanks for providing an open source library for Torch

Jianguo99 commented 1 week ago

Hello, Emir, Thanks for your correction and suggestion.

  1. We have revised the code of ConfTr in the "development" branch. This month, we will release a new version of TorchCP, which will revise this problem.
  2. In the "example" folder of TorchCP, we provide a tutorial for TorchCP, which can help users quickly deploy TorchCP with their own dataset.

Thanks for your issue again. Please let us know if you have more questions.

Best regards, authors

emirceyani commented 1 week ago

Thanks a lot and sure!