search

EPRV3EvidenceChallenge / Inputs

Input Data & Model for the EPRV3 Evidence Challenge - Start Here
MIT License
11 stars 9 forks source link

MULTINEST, multi-run MULTINEST and VB-importance sampling results from new priors #18

Closed SurangkhanaRukdee closed 7 years ago

j-faria commented 7 years ago

@SurangkhanaRukdee, these results are with the broad priors (1.25<P<1e4, e=rayleigh)? I'm getting very different results, by many orders of magnitude, and it seems that @vineshrajpaul is as well from what I see in #17.

vmaguirerajpaul commented 7 years ago

I noticed this as well. Of course I am not sure that my own results are in the right ballpark; however, they are at least fairly similar to the results I got with the old priors (and similar I think to TeamPUC's old results), which seems sensible since the likelihoods are unchanged, and the changes to the priors were relatively minor.

JohannesBuchner commented 7 years ago

there is a problem with different george versions between computers, so these results are wrong... sigh We will try to redo them over the weekend.

benelson commented 7 years ago

It would be nice if we had a unit test for the 1-planet model. But not everyone is parameterizing orbital phase in the same way. It doesn't matter for the marginal likelihood, but we should make sure everyone is implementing the priors and Keplerian model correctly.

Eric proposed one of two options: 1. have the unit test be based on the phase that maximizes the log-likelihood if the other parameters are held fixed or 2. have it be based in the integral over all phases.

What does everyone else think?

On Fri, Sep 15, 2017 at 8:56 AM, Johannes Buchner notifications@github.com wrote:

there is a problem with different george versions between computers, so these results are wrong... sigh We will try to redo them over the weekend.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/EPRV3EvidenceChallenge/Inputs/pull/18#issuecomment-329790169, or mute the thread https://github.com/notifications/unsubscribe-auth/AGSUSxA5ZlF2b_SXOXRMNm_wx0UYRg_Nks5sioIcgaJpZM4PYU33 .

JohannesBuchner commented 7 years ago

We could also fix V, sigma_j, P, K, e and find the maximum likelihood over the other two parameters, which should be pretty trivial?

j-faria commented 7 years ago

For the purpose of testing, isn't it enough to provide (many) reference values for the prior and likelihood evaluated at different parameter values?

JohannesBuchner commented 7 years ago

@j-faria I think we (actually ExoFit) parametrizes the argument of pericenter and/or mean anomaly differently, which does not matter for the evidence but it does for this.

j-faria commented 7 years ago

Ok I understand that. But note that most of the bugs (at least those I saw in the discussions) were in the kernel and/or priors so extensive tests for the 0-planet model likelihood and the priors would suffice.

JohannesBuchner commented 7 years ago

A test for the correct implementation of the prior, which works for both pdf and cdf implementation methods, could be to draw 100000 points from it and compute mean and std in each dimension?

j-faria commented 7 years ago

Yes, I think so. Only caveat is we cannot test that to floating point precision. We could also provide, the actual pdf, logpdf and cdf evaluated at a set of values,

output1 = period_prior.pdf( linspace(1.25, 1e4, size=1000) )
output2 = period_prior.logpdf( linspace(1.25, 1e4, size=1000) )
output3 = period_prior.cdf( linspace(0, 1, size=100) )

for example.