d-ailin / GDN

Implementation code for the paper "Graph Neural Network-Based Anomaly Detection in Multivariate Time Series" (AAAI 2021)
MIT License
467 stars 140 forks source link

Reproduce the results for SWaT dataset #29

Closed giorgigona closed 2 years ago

giorgigona commented 2 years ago

I preprocessed SWaT dataset according to scripts/process_swat.py

I set the hyperparameters for model training according to your comment: https://github.com/d-ailin/GDN/issues/4#issuecomment-876086591

I got significantly different scores compared to paper results. F1 score: 0.7699076110866696 precision: 0.9727626459143969 recall: 0.6371745858365192

Can you confirm that I set the correct hyperparameters?


seed=6799
BATCH_SIZE=16
SLIDE_WIN=5
dim=64
out_layer_num=1
SLIDE_STRIDE=1
topk=15
out_layer_inter_dim=128
val_ratio=0.1
decay=0
EPOCH=50
report='best'
d-ailin commented 2 years ago

I think the hyperparameters look fine.

  1. Could you help check your processed data with the given part of the processed data provided in the given link (https://drive.google.com/drive/folders/1_4TlatKh-f7QhstaaY7YTSCs8D4ywbWc)?
  2. If it is fine, could you run the program with the same seed again for several times to see. I guess it could be some issue similar in the previous issues, #17. Besides, we obtained our results on GPU, though it should be expected to behave the same as on CPU with same seed, but it is not guaranteed as there might exist some libraries that might invite randomness in their implementation. So for this case, you could also just try with some other seeds.

Hope it helps :)

giorgigona commented 2 years ago

It helped thanks!