search

astrosonnen / allZeLenses

Cosmology with every single gravitational lens ever.
0 stars 0 forks source link

Spherical Cow / SL2S Paper V model #7

Closed astrosonnen closed 8 years ago

astrosonnen commented 9 years ago

I generated an SL2S-IV-like sample of time-delay lenses and fitted it with the same model used to generate the mock. The inference on H0 is biased. The issue might be due to the way I sample over the individual systems and/or the way I calculate the integrals over the individual lens parameters.

drphilmarshall commented 9 years ago

Maybe there is some interim/conditional PDF truncation going on? I feel like that could be an issue...

astrosonnen commented 9 years ago

I think I narrowed down the origin of the problem. The bias starts occurring when sampling over the source position. If I use the same set of mock lenses and I do the inference fitting only the Einstein radii instead of the image positions, the bug disappears. Now why does the marginalization over the source position introduce a bias in the mean dark matter mass?

astrosonnen commented 9 years ago

By the way, the conditional PDF of the individual systems is being truncated, but the probability distribution in the individual parameters given the hyper-parameters is renormalized accordingly during the hierarchical inference, so it is not an issue.

drphilmarshall commented 9 years ago

​Are you solving the lens equation to predict image positions, or tracing back image positions to estimate the source position? The latter is known to introduce a bias, even if you trace back the image position uncertainties via the magnification matrix​ as well.

On Wed, Mar 4, 2015 at 4:54 PM, astrosonnen notifications@github.com wrote:

I think I narrowed down the origin of the problem. The bias starts occurring when sampling over the source position. If I use the same set of mock lenses and I do the inference fitting only the Einstein radii instead of the image positions, the bug disappears. Now why does the marginalization over the source position introduce a bias in the mean dark matter mass?

— Reply to this email directly or view it on GitHub https://github.com/astrosonnen/allZeLenses/issues/7#issuecomment-77284554 .

astrosonnen commented 9 years ago

I am solving the lens equation. The source position is another free parameter of the model. I am assuming a flat prior in the square of the distance from the optical axis (so that it is uniform in 2d), the same used to generate the mock, and then I marginalize over the source position when doing the hierarchical inference.

On Wed, 4 Mar 2015, Phil Marshall wrote:

​Are you solving the lens equation to predict image positions, or tracing back image positions to estimate the source position? The latter is known to introduce a bias, even if you trace back the image position uncertainties via the magnification matrix​ as well.

On Wed, Mar 4, 2015 at 4:54 PM, astrosonnen notifications@github.com wrote:

I think I narrowed down the origin of the problem. The bias starts occurring when sampling over the source position. If I use the same set of mock lenses and I do the inference fitting only the Einstein radii instead of the image positions, the bug disappears. Now why does the marginalization over the source position introduce a bias in the mean dark matter mass?

— Reply to this email directly or view it on GitHub https://github.com/astrosonnen/allZeLenses/issues/7#issuecomment-77284554

.

— Reply to this email directly or view it on GitHub.[AGYG6mbQA6xacGnJJVmMRE3SYGiQndL3ks5nx6HXgaJpZM4Dh3hl.gif]

astrosonnen commented 9 years ago

Let's do this. Let's keep the model with the bug for the moment, and let's use it to do our first posterior predictive check with it. Presumably it will fail, maybe we can learn something more from it.

drphilmarshall commented 9 years ago

Good plan. We can compare models that predict the data well, with models that are close to the truth.

Your comparison with the no-source-parameters model seems to show that the priors for our other model parameters are OK. Maybe the issue is the joint PDF for the source position and (at least) one other parameter. Is that parameter the density profile slope?

On Thu, Mar 12, 2015 at 2:54 PM, astrosonnen notifications@github.com wrote:

Let's do this. Let's keep the model with the bug for the moment, and let's use it to do our first posterior predictive check with it. Presumably it will fail, maybe we can learn something more from it.

— Reply to this email directly or view it on GitHub https://github.com/astrosonnen/allZeLenses/issues/7#issuecomment-78641808 .

astrosonnen commented 9 years ago

I think I've found the issue. The prior on the source position should be flat between the center and the radial caustic. The caustic changes with the model parameters, so the prior must be updated as well. If I do that, things seem to finally work...

drphilmarshall commented 9 years ago

I guess this prior follows from the assumption "the object is a strong lens" - but its interesting that it has to be applied so rigorously! Which way did the bias go, do you know? Was it due to truncating the source position, or allowing it to range too widely? In any case - well caught!

On Fri, Apr 3, 2015 at 5:16 PM, astrosonnen notifications@github.com wrote:

I think I've found the issue. The prior on the source position should be flat between the center and the radial caustic. The caustic changes with the model parameters, so the prior must be updated as well. If I do that, things seem to finally work...

— Reply to this email directly or view it on GitHub https://github.com/astrosonnen/allZeLenses/issues/7#issuecomment-89471388 .

astrosonnen commented 9 years ago

I think I was allowing for too large values of the source position. It makes little to no difference on the individual lens, but in the hierarchical inference it gave a 2-sigma bias on the dark matter mass (and therefore on the density slope and the cosmology).

On 04/06/2015 09:28 AM, Phil Marshall wrote:

I guess this prior follows from the assumption "the object is a strong lens" - but its interesting that it has to be applied so rigorously! Which way did the bias go, do you know? Was it due to truncating the source position, or allowing it to range too widely? In any case - well caught!

On Fri, Apr 3, 2015 at 5:16 PM, astrosonnen notifications@github.com wrote:

I think I've found the issue. The prior on the source position should be flat between the center and the radial caustic. The caustic changes with the model parameters, so the prior must be updated as well. If I do that, things seem to finally work...

— Reply to this email directly or view it on GitHub

https://github.com/astrosonnen/allZeLenses/issues/7#issuecomment-89471388 .

— Reply to this email directly or view it on GitHub https://github.com/astrosonnen/allZeLenses/issues/7#issuecomment-90133399.

drphilmarshall commented 8 years ago

While we work on the second checkbox of this epic above, we also need a notebook to keep track of it. That should be the move we make right after we agree on the PGM we are talking about.

astrosonnen commented 8 years ago

The inference appears to be accurate now, as it should. All bugs found so far are documented here

drphilmarshall commented 8 years ago

Excellent! I see the notebook that is linked to above only contains the "problem formulation", though - I think this notebook should be named after the model we have defined, and have a results section as well as a PGM section. Show us the corner plots! :-)