search

Leezekun / ViTST

[NeurIPS 2023] The official repo for the paper: "Time Series as Images: Vision Transformer for Irregularly Sampled Time Series"."
https://arxiv.org/abs/2303.12799
101 stars 5 forks source link

Reported metrics for RAINDROP method is very different from that in the original paper RAINDROP #14

Closed Xiongbenyang closed 1 month ago

Xiongbenyang commented 1 month ago

Hi,

It seems the results of the compared method in Table 1 of the VITST paper, e.g., raindrop is very different from that reported in the original paper of raindrop. Specifically, on the P12 dataset, the reported AUROC and AUPRC is 82.8 and 44.0, while that of the original raindrop paper is 72.1 and 97.0.

I am assuming ViTST is using the same processed P12 dataset as the raindrop paper, including the 5 splits, so I have no idea on the reason why such big performance gap occurs.

Thanks!

Leezekun commented 1 month ago

Hi, thanks for your interest. The difference lies in the labels and prediction targets used.

The Raindrop paper reported results on P12 using a binary label indicating the length of stay in the ICU: a negative label for hospitalization not exceeding 3 days, and a positive label for hospitalization longer than 3 days. In contrast, our results use a label indicating survival, which aligns better with the intended purpose of the data.

Xiongbenyang commented 1 month ago

Got it. Thank you very much for your reply.