Kevin-McIsaac
commented
7 years ago In a recent project I had to find the outliers for a large number of time series. I discovered the typical process is first remove any trend and/or seasonal component to create 'residues'. I used smf.ols residues from a linear model but this could be extended to use ARMA etc.
In many cases the residues will be near normal so you can use a robust measure of dispersion (ie., based on mean/MAD or IQR) to identify the outliers.
While simplistic compared to the proposal above, it worked well in my analysis and is a simple place to start.
I wrote some code for panda Series, created a basic notebook and proposed an enhancment in panda-dev. The recommendation was to put this in statsmodels/
Would you be interested in adding something like this? If so I can rework it to fit better with statsmodels.
josef-pkt
kris-singh
related to #943
943 and related discussion is on how to handle outliers. AFAICS, the main approach for outlier detection and handling for ARMA and similar models are based intervention models, and "oxford" dummy saturation. AFAIK, this ignores the impact of the outlier search on inference.
Soren Johansen and Bent Nielsen have a recent or current sequence of papers that includes asymptotic theory and is closer to the robust literature, e.g.
Johansen, Søren, and Bent Nielsen. 2016. “Asymptotic Theory of Outlier Detection Algorithms for Linear Time Series Regression Models.” Scandinavian Journal of Statistics 43 (2): 321–48. doi:10.1111/sjos.12174.
Johansen, Søren, and Bent Nielsen. 2013. “Outlier Detection in Regression Using an Iterated One-Step Approximation to the Huber-Skip Estimator.” Econometrics 1 (1): 53–70. doi:10.3390/econometrics1010053. http://www.mdpi.com/2225-1146/1/1/53
This is just a reference for now (so we find it again), I have no idea when we will get into this neighborhood. The priority should be to implement the traditional methods for outlier handling in tsa models with focus on prediction and less emphasis on inference. (Quality of forecast standard errors has not been a top priority for anyone either.)
an issue that might be relevant even for traditional methods is the possible difference in convergence or asymptotic distribution depending on one-step WLS, one-step Newton and iterated solutions. see #3273
as an example: unit root tests under outlier removal (just quick skimming of the second paper, 2013, above: it has the Brownian motion based distribution for the AR(1) coefficient for/under unit root as an example.) (related: somewhere I have seen articles on robust VAR and cointegration)