Goodness of fit tests: filling in the gaps

[azev77]

This package already has some nice options.

For general distributions

• [x] Anderson-Darling
• [ ] Cramér–von Mises
• [x] Kolmogorov-Smirnov
• [ ] Kuiper
• [x] Pearson Chi Square
• [ ] Watson U-squared statistic (1961) (Matlab code w/ MIT License)

https://discourse.julialang.org/t/multivariate-normality-tests/8384/5

Tests for univariate normality:

• [ ] Anscombe-Glynn kurtosis test
• [ ] Bonett-Seier test for Geary kurtosis
• [ ] D'Agostino's K-squared test
• [ ] Frosini test for normality
• [ ] Geary test for normality
• [ ] Hegazy-Green test for normality
• [x] Jarque-Bera
• [ ] Jarque-Bera Urzua 1996
• [ ] Jarque-Bera Gel Gastwirth 2008
• [ ] Lilliefors test (based on KS)
• [ ] Minimum Hellinger Distance normality test
• [ ] Spiegelhalter test for normality
• [ ] Weisberg-Bingham test for normality

For specific distributions:

• [ ] Doornik-Hansen test for multivariate normality @andreasnoack code here
• [ ] Baringhaus-Henze (multivariate normal)
• [ ] Shapiro-Wilk (multivariate normal) A lot of work on it https://github.com/JuliaStats/HypothesisTests.jl/pull/124
• [ ] Shapiro–Francia test (multivariate normal)
• [ ] Mardia-Kurtosis/Skewness (multivariate normal) (Mardia, K. V. (1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3), 519-530. doi:10.2307/2334770)
• [ ] distance‐to‐boundary (multivariate uniform)
• [ ] Yang Modares 2017 (multivariate uniform)

These tests can be nicely combined to study goodness of fit:

using Distributions, Random, HypothesisTests;
Cts_Uni = subtypes(ContinuousUnivariateDistribution)
DGP_True = Normal(7,7);
Random.seed!(123);
const ds = rand(DGP_True, 1_000); #Training Data.
Er =[]; D_fit  =[];
for d in Cts_Uni
    println(d)
    try
        dd = "$(d)"   |> Meta.parse |> eval
        D̂ = fit(dd, ds)
        Score = [loglikelihood(D̂, ds),
                OneSampleADTest(ds, D̂)            |> pvalue,
                ApproximateOneSampleKSTest(ds, D̂) |> pvalue,
                ExactOneSampleKSTest(ds, D̂)       |> pvalue,
                #PowerDivergenceTest(ds,lambda=1)  Not working!!!
                JarqueBeraTest(ds)                |> pvalue   #Only Normal 
        ];
        #Score = loglikelihood(D̂, ds) #TODO: compute a better score.
        #push!(Dfit, (D̂, Score))
        push!(D_fit, [d, D̂, Score])
    catch e
        println(e, d)
        #push!(Erfit, d)
        push!(Er, (d,e))
    end
end
Er
D_fit
a=hcat(D_fit...)
M_names =  a[1,:]
M_fit   =  a[2,:]
M_scores = a[3,:]
idx =sortperm(M_scores, rev=true)
Dfit_sort=hcat(M_names[idx], M_fit[idx], sort(M_scores, rev=true) )

Dfit_sort=hcat(M_names[idx], sort(M_scores, rev=true) )

Note the pvalue for AD > ApproxKSTest > ExactKSTest

 Normal           Normal{Float64}(μ=7.18058, σ=7.05703)           [-3372.96, 0.699197, 0.674689, 0.665941, 0.493399]
 Laplace          Laplace{Float64}(μ=7.07809, θ=7.40084)          [-3461.94, 3.92474e-6, 6.98194e-5, 6.59328e-5, 0.493399]
 Cauchy           Cauchy{Float64}(μ=7.07809, σ=4.9397)            [-3569.55, 6.0e-7, 2.48653e-8, 2.25733e-8, 0.493399]
 Uniform          Uniform{Float64}(a=-15.0028, b=26.9612)         [-3736.81, 6.0e-7, 5.30718e-44, 1.47801e-44, 0.493399]
 Exponential      Exponential{Float64}(θ=7.18058)                 [-Inf, 6.0e-7, 7.7836e-22, 5.38715e-22, 0.493399]
 Rayleigh         Rayleigh{Float64}(σ=7.11908)                    [-Inf, 6.0e-7, 4.1374e-41, 1.34284e-41, 0.493399]
 InverseGaussian  InverseGaussian{Float64}(μ=7.18058, λ=35.7869)  [-Inf, 6.0e-7, 1.18454e-60, 1.11039e-61, 0.493399]

It would be great if it was possible to automate test properties. This paper does a monte carlo simulation to compare the power of eight tests for normality.

@schrimpf does simulations to explore power here and here.

[PaulSoderlind]

Good ideas. Several of these are both easy and useful. I would probably recommend starting with Kuiper and Pearson's chisquare. Also, Shapiro-Wilk sounds useful, but I am not sure about how tricky it is to code.

[wildart]

There exists a transformation for W statistics that approximates it to normal distribution, see https://link.springer.com/article/10.1007/BF01891203.

[azev77]

@andreasnoack it looks like you have some really nice code for the Doornik-Hansen test for multivariate normality. Would it be possible for it to be ported here?

[andreasnoack]

Sure. I'd be happy to move it here.

[azev77]

Also: PowerLaws.jl: Vuong's (1989) test for non-nested model selection, Clarke's (2007) test for non-nested model selection, Kolmogorov Smirnov test

Pingouin.jl: Levene's test & Bartlett's test for equality of variances Shapiro-Wilk, Shapiro-Francia, Anderson-Darling test Mauchly and JNS test for sphericity Mann-Whitney U Test (= Wilcoxon rank-sum test)

[korilium]

Is there any progress on this?

[nalimilan]

We have included several new tests recently. If you're interested in some of them which are still missing, help is welcome!