search

sharanbngr / blip

A bayesian pipeline for detecting stochastic backgrounds with LISA.
MIT License
16 stars 12 forks source link

Anisotropic response bugfix #122

Closed criswellalexander closed 3 months ago

criswellalexander commented 4 months ago

Removing erroneous factor of 1/4pi from anisotropic responses.

sharanbngr commented 4 months ago

Thanks for finding this bug Alexander. Have you done a sample run with the factor removed to check if recoveries are consistent? Just want to make sure that we are not missing the factor being removed somewhere else in the code or something else that we might have overlooked.

criswellalexander commented 4 months ago

See plots below for consistency check in the data generation.

image

image

I also performed a short test run simulating with the pixel-basis anisotropic method and recovering with the ISGWB analysis and see no issues. The corner and spectral plots are below; the posterior is slightly undersampled and was for a short observing duration/constrained frequency band, but the results are unbiased (which is the main thing). I can try the reverse case as well, if you like, but everything is behaving identically.

corners_all

spectral_fit_detector

spectral_fit_astro

sharanbngr commented 4 months ago

@criswellalexander Sorry for the late reply and thank you for the plots! The other case would be good to check because all of our published recoveries are with the anisotropic/spherical harmonic model. It would be good to check that everything is consistent and that, for e.g. we haven't missed some other factor somewhere else that was accounting for the 4*pi.

criswellalexander commented 4 months ago

Ah, yeah, good idea to cover all the bases. The sph case is consistent, see plots below for an isotropic signal simulated with the sph basis and recovered with the ISGWB analysis.

corners_all

spectral_fit_astro

spectral_fit_detector

I also considered an otherwise identical dataset created with the isotropic routines and recovered with the sph analysis, but this duration is too short to achieve reliable results. It would be good to rerun this case with a 6mo or 1yr duration. That said, while the analysis doesn't produce good quality results and shows some bias, the bias it does show is << 4pi and is a known phenomena for any of the anisotropic analyses without a sufficient duration to "see" the whole sky as the sensitive parts of the LISA antenna patterns sweep across it.

image (the truevals aren't showing up because I forgot to alias the model and injection names, but the spectral values should be the same as above)

image

image

As you can see, the inferred spatial distribution is far from isotropic, hence the bias: image

I've submitted a 6 month run and will provide those results when it finishes.

criswellalexander commented 4 months ago

The 6 month run finished; the results are still poor due to the model misspecification, but they continue to be consistent with having no additional factor of 4pi floating around --- if we did, I would expect to see a large bias in the astrophysical reconstruction. See plots, below. image

image

image

image

As an aside, it is interesting to note that these results (with the numpyro sampler) cling to those bimodalities a bit more that what we see with dynesty, as handling multimodalities is a strength of the latter approach.

sharanbngr commented 4 months ago

Thanks Alexander! By model misspecification, do you just mean the fact that the run is anisotropic while the signal is isotropic? In that case, the posterior should still eventually converge to the right values no? Anyway, looks like this numpyro chain hasn't converged, so can restart and continue the sampling a bit more please? Sorry for assigning more work to you but I think it would be good to see one successfully converged run to convince ourselves we didn't miss anything else haha.

criswellalexander commented 4 months ago

I've set it running for another 15k samples. My worry, though, is that we've never actually looked closely at the behavior of the anisotropic search applied to an isotropic background. From the corner plots, the sampler is converging to extremal (and opposite) values of the sph. harm. component amplitudes, presumably trying to cancel each other out; I don't necessarily see that behavior damping out and converging to zero in each. I'll also consider another more in-depth run here while I'm finishing up some dev work for the re-analyses for my thesis, but after that I'll need to be allocating resources to those analyses.

Overall, the behavior of the sph search for an isotropic signal is probably worth investigating as that will be important to understand, but I'm at this point I'm confident that we're not seeing any effects on the order of the 4pi bias. I appreciate the desire for a nice converged run, though :)

sharanbngr commented 4 months ago

@criswellalexander yeah that's a fair argument. And the spectral plots are convincing at any rate. After thinking about it, I do feel comfortable merging this PR.

As for doing anisotropic analysis with an isotropic background, I'm fairly sure I did some way back when we were developing the method haha, and they were consistent with what we expect. The behavior you're seeing could purely be an issue with numpyro. I've been finding that depending on the number of walkers and dimensionality, it's sometimes not very good at converging.