arundo / adtk

A Python toolkit for rule-based/unsupervised anomaly detection in time series
https://adtk.readthedocs.io
Mozilla Public License 2.0
1.1k stars 146 forks source link

STLDecomposition Implementation #13

Open yangyu-1 opened 4 years ago

yangyu-1 commented 4 years ago

Hello, First of all thank you for open sourcing the package; it is excellently written.

One potential issue I've noticed: the STL Decomposition seems to be implemented using a rolling window method for detrending data, not the Loess method. Here is a code snippet from transformer_1d.py:

def _remove_trend(self, s):
        s_trend = s.rolling(
            window=(self.freq_ if self.freq_ % 2 else self.freq_ + 1),
            center=True,
        ).mean()
        return s - s_trend

Perhaps I'm missing something? The current implementation seems to be what Hyndman calls Classical Decomposition, and not STL. There is a python library that does STL decompose and it is relatively lightweight. And if what I'm describing is indeed the behavior, can I work on implementing STL with Loess?

Any feedback would be greatly appreciated, Thank you

tailaiw commented 4 years ago

@yy910616 Thank you for raising the issue and sorry for the late response (we were on holiday these days in US). I will look into this issue this week and get back to you.

tailaiw commented 4 years ago

@yy910616 I think you are right. I appreciate that you raised the problem. I'm gonna work on this and hopefully have it in the next release. Will keep you updated when a PR is ready to be reviewed.

tailaiw commented 4 years ago

@yy910616 I just realize you proposed to work on the implementation. Sorry for missing it. You are definitely welcomed to join the contributors. I opened a WIP PR #18 for this issue.

yangyu-1 commented 4 years ago

@tailaiw Awesome! I have pretty limited experience with open source but I'll definitely give it a shot. Thanks for opening the PR, I've been doing some research on STL and I'll continue the discussion in the PR.

tailaiw commented 4 years ago

After discussion in #18, we decided to remove the wrong implementation of STL decomposition, and merge two implementations of classic decomposition into the adtk.transformer.ClassicSeasonalDecomposition. We decided to hold on implementation of a new STL decomposition until statsmodels' release with its first implementation of STL algorithm.

tailaiw commented 4 years ago

@yy910616 i just noticed that statsmodels v0.11 has been released.

yangyu-1 commented 4 years ago

Nice. I'm looking into this. Thanks for pointing it out.