EPRV3EvidenceChallenge / Inputs

Input Data & Model for the EPRV3 Evidence Challenge - Start Here
MIT License
11 stars 9 forks source link

Planet ordering & evidences #12

Closed vmaguirerajpaul closed 7 years ago

vmaguirerajpaul commented 7 years ago

Of possible relevance to the 1-3 planet models: does imposing period ordering affect the computed evidences compared to the case where no ordering is imposed? (Sorry if this is a naive question or one that has already been addressed elsewhere!)

JohannesBuchner commented 7 years ago

I have thought about this a little bit. With MultiNest we are using a feature where if a sample is drawn in an unwanted region of the parameter space, the loglikelihood can return a small value (e.g. <-1e90) and MultiNest will just draw a new value, and pretend that space did not exist. We use this when the planets are not sorted (in period).

If you think of the space period1 vs period2 and imagine the line period1=period2, removes the parameter space on the bottom right (period1>period2). To compute Z, both halves would have the same integral over their space, and would be added with weight 1/2 (size of the prior space). However, you do not need to multiply the result by 2 if you already assign the size of the prior space to 1 in the calculations instead of 1/2 (n_planets! in general). This depends on how you set up the prior, prior transforms do that (normalise to 1), probability densities with rejections would not, you would have to add a factor.

One difficulty arises when the evidence for 1 planet is similar to that of 2 planets. Then there are (two) solutions P1~=P2 (i.e. very close to the diagonal). These can be hard to identify (because they are near the edge of the parameter space), but may contribute to some fraction to the integral. You can see this here in the percentiles: https://github.com/EPRV3EvidenceChallenge/Inputs/blob/master/results/TeamPUC/multinest-ins-nlive2000-eff0.3/params_2_0001.txt I wrote in this hidden commit conversation https://github.com/EPRV3EvidenceChallenge/Inputs/commit/406b36b94158719f457f677f10b2b05df253e8f4#commitcomment-23572890: "... However, there is still the minor solution that both planets have P=12 or both planets have P=42. So the prob distribution of the low-P one has a distribution mostly around P=12, but with a small solution at P=42 and the high-P one has a small solution around P=12 but most of its distribution is around P=42. The outer percentiles show this."

So to summarize: in principle the computed evidences should not be affected, but the parameter space borders can make difficult non-Gaussian shapes. Depending on the algorithm and data set this can be better than having many modes.

Anyone correct me if I am wrong :)

vmaguirerajpaul commented 7 years ago

Ok, that seems to make sense, thanks! I'm also integrating over unit hyper-cubes and using prior transforms, and indeed it seems my evidence estimates are not sensitive to imposing ordering on periods.

eford commented 7 years ago

Thanks for raising this discussion. I'm pretty sure that some people weren't accounting for this issue properly. During the Wednesday breakout session, we discussed the issue (and there i syour discussion archived here), so everyone (who was in the room) should be aware of it for when they compute updated estimates of the marginal likelihood.

Thanks, Eric

On Aug 16, 2017 4:18 AM, "vineshrajpaul" notifications@github.com wrote:

Ok, that seems to make sense, thanks! I'm also integrating over unit hyper-cubes and using prior transforms, and indeed it seems my evidence estimates are not sensitive to imposing ordering on periods.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/EPRV3EvidenceChallenge/Inputs/issues/12#issuecomment-322699739, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQywropdm4aySt8BkrCgs8YgqFIjhwJks5sYqW1gaJpZM4O4Ndi .

eford commented 7 years ago

Sec 2.3.3 of the model.pdf addresses this issue. I think it's resolved, so I'm closing the issue.