search

thuml / Anomaly-Transformer

About Code release for "Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" (ICLR 2022 Spotlight), https://openreview.net/forum?id=LzQQ89U1qm_
MIT License
733 stars 190 forks source link

questions on validation set and threshold selection algorithm #32

Open carrtesy opened 1 year ago

carrtesy commented 1 year ago

Hello,

I have a question regarding your work.

  1. How validation set are selected here? from dataloader code, validation set seems to be test set. Did I get the codes right?
  2. Anomaly Transformer's thresholding mechanism. Is the model using test dataset rather than validation set to set the threshold? From Appendix H, validation set (which is consisted of normal data only) is used to pick an appropriate threshold. However, the code seems to utilize test dataset for thresholding.
    Did I miss some points?

Thanks a lot for your answer in advance.

xjw-wade commented 1 year ago

I have the same question too, thanks a lot for your answer in advance.

carrtesy commented 1 year ago

This issue seems to be addressed on previous issue. However, I still wonder: (1) If the F1-score reported on the paper use validation set to attain threshold (Implementation details of Section 4. Experiement states that the thredhold $\delta$ is determined using validation dataset.) (2) If the Protocol of threshold selection (appendix H) are guaranteed to provide threshold that are close to the optimal threshold.