search

aewallin / allantools

Allan deviation and related time & frequency statistics library in Python
GNU Lesser General Public License v3.0
216 stars 78 forks source link

Outlier/jump detector functions? #56

Open mihaispetrescu opened 7 years ago

mihaispetrescu commented 7 years ago

Hi,

Great project! I was wondering if there's any interest in adding robust statistics methods to remove outliers or detect frequency jumps.

aewallin commented 7 years ago

Yes, there is already one gap-robust adev function, but more are welcome. Preferrably based on some published papers or established algorithms. Also think about how to test new functions i.e. what datasets to use and how to test that the algorithm is actually working. Then also think about how new functions fit to the API - are they filters that run on the data before calling a standard adev() function or how should it work?

mihaispetrescu commented 7 years ago

There are some algorithms in stable32 that would be useful, and can be ported. I think stable32 uses median absolute deviation for detecting outliers. For detecting jumps, I've used sequential and block averaging, and Riley has a pretty good write-up on pros/cons. It's quite possible they may already be implemented in python/numpy somewhere, but they wouldn't be hard to write up if not. An algorithm like DAVAR can be useful (it may be implemented in your package already, i apologize if i haven't seen it!).

mihaispetrescu commented 7 years ago

Also, I've seen Dr. Tavella papers floating around that work with other techniques.

I'd love to contribute to your project, and look forward to your thought.

aewallin commented 7 years ago

Dynamic adev, median absolute deviation, and jump detection would all be good additions to the library. It's good if there is a paper as a reference (describing the algorithm and maybe giving a sample implementation), some test dataset (real or synthetic) and some reference results to test against (e.g. Stable32, if what it does is documented well enough). If you have time to work on allantools you could also consider finishing the bias-correction work and/or the confidence interval work that is now half-finished until I or someone else has time and energy to finish it.