Closed mobeets closed 8 years ago
can't just shuffle:
say X,Y ~ N(0,1) and you try to predict n draws of X and Y with each other, sample for sample.
well, this will have some error, but if Y instead is just a constant (the mean of X: 0) then the error will be lower.
so this shuffling thing doesn't measure a match in distribution...not even a match in mean.
i'd need a nonparametric comparison of distributions...
this is the two-sample problem
according to larry, simplest way is basically to do kernel density estimation (aka parzen window estimation) on each of the two distributions and then take the L_2 norm of their differences. the bandwidth can be relatively large (see: peter hall paper linked below), which is good for high-d
http://www.jmlr.org/papers/volume13/gretton12a/gretton12a.pdf http://www3.stat.sinica.edu.tw/statistica/oldpdf/a17n412.pdf https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
the smola MMD paper actually has a good discussion of the parzen window method as well
though from smolas, also: "This suggests it is not necessary to solve the more difficult problem of density estimation in high dimensions to do two-sample testing."
Kolmogorov-Smirnov (K-S) too
can you first get a null distribution by shuffling observeds to predict observeds? then compare this to the hypotheses.