search

haskell / statistics

A fast, high quality library for computing with statistics in Haskell.
http://hackage.haskell.org/package/statistics
BSD 2-Clause "Simplified" License
299 stars 67 forks source link

Problems with Student-T distribution #160

Open Shimuuar opened 4 years ago

Shimuuar commented 4 years ago

So far following problems with Student-T distribution were identified:

  1. quantile lose precision for p≈0.5 since x becomes close to 1, while p is near zero

    quantile :: StudentT -> Double -> Double
quantile (StudentT ndf) p
| p >= 0 && p <= 1 =
let x = invIncompleteBeta (0.5 * ndf) 0.5 (2 * min p (1 - p))
in case sqrt $ ndf * (1 - x) / x of
     r | p < 0.5   -> -r
       | otherwise -> r

  2. Both precision and performance suffer greatly for large degrees of freedom. Reason is likely incomplete beta and its inverse perform poorly for large parameters. However distribution becomes close to normal so other approximations could work!

Shimuuar commented 4 years ago

RE 2.

Quick experimentation with mpmath shows that precision becomes worse for larger NDF. (N of ulps of error proportional to NDF). For very large N mpmath fails itself which I think doesn't bring any good news for algorithm being used...