search

zhhlee / InterFusion

KDD 2021: Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding
MIT License
197 stars 46 forks source link

The strategy for caculate f1 score #15

Closed Conearth closed 2 years ago

Conearth commented 2 years ago

Hi, this is an amazing job! Thx for your attention. I have a few questions about the strategy for caculate f1 score. For ASD and SMD, it seems that you trained your model and caculate the result once on a single entity like "omi-x" or "machine-x-x". So how to caculate the final score for the whole dataset? And is there any possible to train the model on the all entities together? Have you tried? I'll appreciate it if you can help out.

zhhlee commented 2 years ago

Yes, we train a model for each single entity in ASD and SMD, and the final score for the whole dataset is the average F1 score for each entity. We did not train the model on all entities together. One entity represents the monitoring metrics of a server, and we train a model to detect anomalies on that server. Different entities may have different normal patterns. It might be hard for the model to learn such patterns together. However, you may try to train the model on all the entities together, just using the training data for all entities as the training set. Maybe a mixture of prior distribution is needed to learn the diverse patterns.

Conearth commented 2 years ago

Many thx.