search

blei-lab / edward

A probabilistic programming language in TensorFlow. Deep generative models, variational inference.
http://edwardlib.org
Other
4.83k stars 759 forks source link

leave one out cross validation #109

Open dustinvtran opened 8 years ago

dustinvtran commented 8 years ago

Following the explanation in Vehtari, Gelman, Gabry (2016), this is a very simple wrapper around ed.evaluate(). The only algorithmic change is importance sampling, so that we can use samples from p(z | y) so it is as if they came from the leave-one-out posterior p(z | y_{-i}).

kaushik94 commented 6 years ago

I have read the research paper on LOOCV as mentioned in the issue and I have a question regarding implementation of the wrapper. It is clear that when we run evaluate by removing y_i in each run for i from 0 to N, we get a leave-one-out predictive distribution. Correct me if I am wrong here, I believe since this process is expensive we do an importance sampling on the y_pred array so we can somehow easily calculate the approximate loocv predictive distribution. But I am having trouble coming up with a way to calculate importance ratios in the code. more specifically calculating the probability distribution of p(θ|y). Can you help me out here? The question might come out as silly. Sorry I am not that good with advanced math. Your inputs would be insightful.