search

ml-stat-Sustech / TorchCP

A Python toolbox for conformal prediction research on deep learning models, using PyTorch.
GNU Lesser General Public License v3.0
193 stars 26 forks source link

Non-Scalar output #26

Closed Naor-F closed 6 days ago

Naor-F commented 2 weeks ago

Hello,

I'm attempting to utilize the toolbox for a non-scalar output regression task. Do you have an example I could refer to?

Thanks,

Jianguo99 commented 1 week ago

Dear Naor-F,

For vector-valued models, you can apply CP to each element individually.

Please let me know if you have any further questions or if there is anything else I can assist you with.

Best regards, authors

Naor-F commented 1 week ago

Hi,

Thank you for your response. Allow me to clarify the question:

In the case of a regression task, for instance, where I have a time series input X and the output Y represents a prediction of the input for a future period, how can I achieve this?

I think using CP on each element individually not be suitable for solving this problem

Thanks, Naor-F

Jianguo99 commented 1 week ago

Dear Naor-F,

Thank you for your message. Could you please clarify whether the length of the predictive period is fixed? If it is, we can apply CP for the same future time point.

Please let me know if my understanding is incorrect. I look forward to your response.

Best regards, authors

Naor-F commented 1 week ago

Hi,

The input consists of N points, and the output consists of M points representing future time: t+1, t+2, ..., t+M. I am predicting M points into the future.

You recommend applying CP to each point separately.

Best regards,

Jianguo99 commented 1 week ago

Dear Naor-F,

I think your problem setup should be using the first N moments to predict the subsequent M moments. With different inputs, the outputs might have overlapping parts. Therefore, it is unreasonable to apply CP to each output value individually.

If this is the case, you might try to average the overlapping parts to implement the standard CP algorithm, including the calibration process and the prediction process (This is my guess and might be difficult to satisfy valid coverage).

Finally, my key suggestion is to run a numerical experiment and see if you might gain additional insights. Looking forward to your response.

Best regards, authors

Naor-F commented 1 week ago

Hi,

Thank you very much for taking time to think about this. Very appreciated.

I will dive into this topic, and see what is the right approach for this kind of problem.

Regards,

On Thu, Jul 4, 2024, 20:41 Jianguo Huang @.***> wrote:

Dear Naor-F,

I think your problem setup should be using the first N moments to predict the subsequent M moments. With different inputs, the outputs might have overlapping parts. Therefore, it is unreasonable to apply CP to each output value individually.

If this is the case, you might try to average the overlapping parts to implement the standard CP algorithm, including the calibration process and the prediction process (This is my guess and might be difficult to satisfy valid coverage).

Finally, my key suggestion is to run a numerical experiment and see if you might gain additional insights. Looking forward to your response.

Best regards, authors

— Reply to this email directly, view it on GitHub https://github.com/ml-stat-Sustech/TorchCP/issues/26#issuecomment-2209395945, or unsubscribe https://github.com/notifications/unsubscribe-auth/BJOYAX7AU35JY6QZX62CDOTZKWCMVAVCNFSM6AAAAABKHEA3RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBZGM4TKOJUGU . You are receiving this because you authored the thread.Message ID: @.***>