search

jlouis / eministat

Port of Poul-Henning Kamp's ministat to the Erlang world
Other
99 stars 9 forks source link

Numerical stability #3

Open jlouis opened 8 years ago

jlouis commented 8 years ago

We could opt for using summation and processing with better numerical stability.

seriyps commented 5 years ago

I found that under some circumstances (measured values in a set are very close or the same big floats), std_dev calculation may fail because variance becomes negative. I guess it's somehow related to instability? Should some other "variance" calculation algorithm be implemented? https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Na%C3%AFve_algorithm