Design metric for evaluating time series anomaly detection

[zer0n]

• Study object detection metric
• Design TS anomaly detection metric
• Explain to Mobius team about the metric and the label protocol

[satyanshukla]

Evaluation Metrics for different cases in anomaly detection:

1. Point-wise Labeling: Area under ROC curve, Precision and Recall
2. Region-wise Labeling:
   • Mean Average Precision (mAP) : same as used in Object Detection
   • Weighted mAP: In order to give more weightage to early detection (using a sigmoid like function)
3. Fixed Length Intervals: Area under ROC Curve

[zer0n]

On point-wise labeling: if we predict an anomaly 1 time step earlier or later, would that be counted as a misclassification? That doesn't seem right.

On fixed-length interval: I don't understand the AUC metric. What would it mean to be a false positive or a false negative?

On weighted mAP: can you provide the mathematical formulation for the metric? In particular, I'm curious how you apply the Sigmoid function.

[satyanshukla]

1. Regarding pointwise labeling, we would have some kind of confidence (could be probability or anomaly score) at each point of time series and then we can calculate the area under ROC score by varying the confidence threshold.
2. Fixed-interval labeling would be similar to above (1) in a way that here there would be an anomaly score for each interval and then by varying the threshold we can calculate AUC. False positive would be when non-anomaly windows are classified as an anomaly. We can treat each interval as a point and do classification. This makes more sense as compared to pointwise labeling.
3. Weighted mAP: Weighted_mAP.pdf