search

dfm / emcee

The Python ensemble sampling toolkit for affine-invariant MCMC
https://emcee.readthedocs.io
MIT License
1.47k stars 430 forks source link

Correct way to calculate the model evidence #348

Open cloud182 opened 4 years ago

cloud182 commented 4 years ago

Hi everyone,

I'm using an EnsembleSampler to fit a series of models to the data. For each model, I'm interested in an estimate of the best fitting parameters, but I'd also like to calculate the odds ratios for all the models in order to understand which one is better describing the data. To do so I need to recover the evidence, i.e. the integral of the un-normalized posterior. Is there a way to recover this information from the output of run_mcmc() when using an EnsembleSampler? I know that PTSampler can do it but I saw that it was moved to another package. Since I'm working on a pre-existing code, I'd prefer not to adapt it to use the new ptemcee package if there is a reasonable way to get these data using the current implementation of the algorithm.

Thanks in advance for the help,

Cheers, Enrico

dfm commented 4 years ago

Unfortunately this isn't possible with vanilla emcee. You could pin your requirements to emcee < 3.0, but there were always issues with the PTSampler so I'd probably recommend switching to something more up-to-date like dynesty if you need to do this!

cloud182 commented 4 years ago

Hi, thanks for the fast reply!

I saw that it is possible to recover the value of the posterior at the end of the process for each point in the chains. Even if it is not possible to get the exact value for the integral, wouldn't taking the mean of these value to be an estimate of the integral, except for a multiplicative factor? I'm not an expert on this, I just want to be sure before committing to significantly change the code.

Thanks again, Enrico

Il giorno ven 10 lug 2020 alle ore 15:51 Dan Foreman-Mackey < notifications@github.com> ha scritto:

Unfortunately this isn't possible with vanilla emcee. You could pin your requirements to emcee < 3.0, but there were always issues with the PTSampler so I'd probably recommend switching to something more up-to-date like dynesty https://dynesty.readthedocs.io/ if you need to do this!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dfm/emcee/issues/348#issuecomment-656860976, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5DOBNZIPAXMWCLRPN5EPLR25WMBANCNFSM4OW2WU2A .

davidwhogg commented 4 years ago

This will not work in general. If you look at the integral you are trying to do, and the integral that is implicitly being done by averaging over samples, you will find that they are very different integrals.

Modifying emcee to deliver a marginalized likelihood would be a big project. It's possible that you could make some iterated emcee method that would implement what's here https://arxiv.org/abs/1401.6128 but I would not recommend it.

cloud182 commented 4 years ago

OK, thank you very much for your help! I'll look into the dynesty package!

Cheers, Enrico

Il giorno dom 12 lug 2020 alle ore 09:13 David W. Hogg < notifications@github.com> ha scritto:

This will not work in general. If you look at the integral you are trying to do, and the integral that is implicitly being done by averaging over samples, you will find that they are very different integrals.

Modifying emcee to deliver a marginalized likelihood would be a big project. It's possible that you could make some iterated emcee method that would implement what's here https://arxiv.org/abs/1401.6128 but I would not recommend it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dfm/emcee/issues/348#issuecomment-657220121, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5DOBK3IPRL3YOOJ4XVFXLR3GZF7ANCNFSM4OW2WU2A .

holderbp commented 2 years ago

David: Sorry to open an old thread, and if the answer to my question is obvious (I will look at dynesty). Is there some intrinsic* problem with using the samples from emcee to calculate marginal likelihoods (and then Bayes factors)? I've adapted your line-fitting tutorial to produce data from one of two error models:

yerr^2 = sigma^2 (parameters: m, b, sigma)

OR

yerr(i)^2 = sigma^2 + sigma_f(i)^2 w/ sigma_f(i) ~ N(0, f*y(i)) (parameters: m, b, sigma, f)

and then performed MCMC with each assumption. The marginal likelihoods (calculated, e.g., by the harmonic mean of the likelihood using "flat_samples" ... although I'd like eventually to use better methods) are reasonable at least in so far as the Bayes factors are large and in favor of using the model hypothesis from which the data was generated.