search

khundman / telemanom

A framework for using LSTMs to detect anomalies in multivariate time series data. Includes spacecraft anomaly data and experiments from the Mars Science Laboratory and SMAP missions.
https://arxiv.org/abs/1802.04431
Other
1.02k stars 248 forks source link

overall f0.5score calculation #36

Closed astha-chem closed 3 years ago

astha-chem commented 4 years ago

Hello @khundman,

Thanks for the paper and for making the code public. I have a question whose answer I couldn't find in the repo.

Table 2 in the paper shows precision, recall and f0.5 score for the overall dataset. How are the results from multiple channels combined to give a single precision and recall for each dataset? Is the f0.5 score here calculated based on the reported precision and recall?

Am I correct in understanding that f0.5 refers to f beta score with beta = 0.5, meaning that more weight is given to precision? In some cases the f0.5 score is lower than both precision and recall, eg. Non-Parametric w/ Pruning (p = 0.13) SMAP. For precision = recall = 85.5%, shouldn't fbeta score also be 85.5% for any value of beta?

selimfirat commented 4 years ago

Thank you @khundman for such wonderful work and for sharing it! I wonder the same question. Any updates on this?

khundman commented 3 years ago

Hi @astha-chem your understanding is correct, this is an error in the paper. The reported precision and recall numbers are accurate, but the reported f0.5 score is not.