search

khundman / telemanom

A framework for using LSTMs to detect anomalies in multivariate time series data. Includes spacecraft anomaly data and experiments from the Mars Science Laboratory and SMAP missions.
https://arxiv.org/abs/1802.04431
Other
1.02k stars 248 forks source link

About reproducing #41

Closed selimfirat closed 3 years ago

selimfirat commented 4 years ago

Thank you for sharing this wonderful work! In fact, I have succeeded to reproduce with higher scores :) This situation makes me wonder why this happens. Was there any project/data update or am I doing something wrong?

I followed the "To run with local or virtual environment" instructions in README.py without changing config.yaml since it says "to recreate the experiment from paper, leave as is". I obtain an absolute 3% more recall.

I obtain 0.87 precision and 0.83 recall, whereas in the paper, it is 87.5% precision and 80.0% recall. Screen Shot 2020-07-19 at 00 37 39

khundman commented 3 years ago

@selimfirat thanks for the note, I was able to reproduce this as well, and the paper is incorrect for this particular experiment. I plan to issue a revised version of the paper with this correction and another that has been found, but not sure on timing of when I'll get to that.