fsaad
commented
6 years ago Related: https://github.com/probcomp/cgpm/issues/14
Related: https://github.com/posterior/goftests
Thoughts:
The label Research is indeed an appropriate one, and very interesting! The statistical problem this ticket touches upon is "goodness-of-fit" testing: are samples X drawn from distribution F? The majority of frequentist goodness-of-fit tests in the continuous setting, Kolmogorov-Smirnov, Cramer-von Mises, Anderson-Darling, are (i) univariate, and (ii) based on the CDF representation of the distribution. While the literature on non-parametric techniques for GOF testing is vast (can refer to references if interested, including recent NPB techniques), I do not know of any other methods which are routinely used as statistical tests in related fields.
Here is one idea for the univariate setting, using the differential entropy as a test-statistic:
- Generate i.i.d samples X1, X2, ... Xn ~ \simulate();
- Estimate the differential entropy H(X) = \sum_{i=1}^{n}-\logpdf(xi)) using Monte Carlo;
- Compute the entropy "exactly" using quadrature;
- If \simulate agrees with \logpdf, the MC entropy should approach the quadrature truth entropy.
It should be possible to compute large sample error bounds of the MC estimator, and obtain confidence intervals.
Currently, most of the built-in probabilistic primitives only have their density functions tested (at a test point). The samplers are untested. It should be possible to develop a standard testing scheme that gives confidence that sampler conforms to the corresponding density function, possibly by estimating KL divergences. At a high-level, the design could continue sampling (gaining more confidence in the fidelity of the sampler) until either a sufficient level of confidence is reached, or a timeout is reached (where exceeding the timeout cause a test failure).