Bayesian Evidence Calculation

[bjfultn]

Add Bayesian evidence calculation for robust model comparison.

[plavchan]

Bump. - Peter Plavchan

[bjfultn]

Did we decide that we couldn't do this @smmills?

[smmills]

I think one could do this, but it's non-trivial. I am un-assigning myself though due to time constraints.

[bjfultn]

@smmills Do you remember what the primary challenge here was? Perhaps ensuring that all of our priors are properly normalized?

[smmills]

While that is necessary, I think there are additional technical challenges. Numerically computing the integral of the likelihood times the prior over all space may be challenging when the volume which the likelihood is "significant" in is extremely small compared to the prior, not to mention the likelihood surface is not necessarily smooth and Gaussian for RVs. A conventional MCMC samples only a small subset of the space of the integral near the max likelihood, converting this into an approximation of the integral may suffer from numerical instability (e.g., with the harmonic mean method or the) or make assumptions which we know are far from true (e.g. the Gaussian/Laplace posterior approximation).

[smmills]

I think that's why people use things like multinest. But attaching radvel to multinest instead of using emcee is also non-trivial.

[bjfultn]

OK, thanks for putting this into the record.

Multinest might be an option. I plan on overhauling the MCMC with a version 2 release sometime soon. I was planning on using emcee3, pymc3 (or maybe wait for pymc4), but I can look into multinest as well.

[vandalt]

Hi @bjfultn, are there plans to include evidence calculations and/or nested sampling to RadVel ? I have used dynesty for simple models recently (not RadVel models) and switching between emcee and dynesty was relatively easy. The main task was to rewrite the MCMC prior distributions as prior transforms for nested sampling. For RadVel, I assume this would mean writing an equivalent prior transform compatible with the modelling API for each implemented prior, and also a dynesty sampler similar to the emcee one in radvel.mcmc. I was planning to try implementing this in the next few months to be able to use both MCMC and nested sampling with RadVel model (no idea how much time I will be able to spend on this). Is this something you would be interested in ? This would make a second sampler to maintain, so I would understand if you prefer not to include it in RadVel directly (I could still give it a try and release it separately if it works, in that case).

[bjfultn]

This sounds great and I'd love if you are able to work on it and eventually submit some pull requests! The main issue we ran into when looking at this problem a few years ago is that some of our priors are not finite and forcing them to be finite might have an affect on the evidence computation.