search

maxjcohen / transformer

Implementation of Transformer model (originally from Attention is All You Need) applied to Time Series.
https://timeseriestransformer.readthedocs.io/en/latest/
GNU General Public License v3.0
842 stars 165 forks source link

Possibility for time series anomaly detection? #52

Closed sdodingnan closed 2 years ago

sdodingnan commented 2 years ago

Thank you for your outstanding work. I would like to know whether the Transformer can be applied to unsupervised time-series anomaly detection tasks, similar to Seq2Seq, Autoencoder model based on reconstruction. If possible, Where does this code need to be changed?
In addition, there is one point in your code that I don’t understand, why the decoder's inputs is the encoder's outputs, it should only be used as memory , Looking forward to your reply!

maxjcohen commented 2 years ago

Hi,

The transformer defined here acts as a model with takes an input and spits out an output. To the best of my knowledge, the input and output vector are fixed, but that shouldn't prevent you from training it on a reconstruction task, or other unsupervised tasks. I encourage you to write your own training loop, you can start with this generic function I used in this repo. It's a bit hand made, mostly for demonstration purposes, but there's nothing more than that. https://github.com/maxjcohen/transformer/blob/bea1c92a336583d28eb5869be312055747b67c73/src/utils/search.py#L11-L39

For the second point, please see #41 .

sdodingnan commented 2 years ago

I got it, thanks. But, how did you choose the seq_length in model? In my experiment, I found that training with a long sequence has a higher reconstruction errors compared to a shorter sequence.

maxjcohen commented 2 years ago

If you're not using one of the chunked attention block (see available modules), reconstruction error might go up with the sequence length as the attention must span over more data.