KiDS likelihood

[srbPhy]

Hi

I need some help in interpreting the chi^2 values for the KiDS dataset (kids450_cf_likelihood_public). My aim is to compare two models (using Planck and KiDS datasets) by comparing the chi-squares of their respective best-fit points. In addition, I also want to see the contribution of each experiment to total chi-square. I was able to do that in MontePython using the inbuilt command --display-each-chi2 and got the wanted result.

But running the same --display-each-chi2 command again (with the same bestfit point) resulted in a different chi^2, which was surprising! It seems that the chi^2 of KiDS dataset changed on calculating it again while that of the other experiments remained the same. I did the same exercise 10-15 times, and found that chi^2 for KiDS can change by as much as 7.

1. Is this an expected behavior of the KiDS likelihood?
2. Digging into the init file, I noticed that it only happens when bootstrap_photoz_errors = True (which is true by default). So I guess I can compare the bestfit chi^2 by turning it off, using bootstrap_photoz_errors = False. Would that be a correct approach?
3. Also, should I be worried that the bestfit point itself may not be correct, as it was calculated with bootstrap_photoz_errors = True? Should I start a fresh run with bootstrap_photoz_errors = False?

Thank you Saurabh

[brinckmann]

Hi Saurabh,

First I should check that you used the flag -f 0 to force the code to not make a jump?

That said, if I remember correctly, that particular likelihood has an internal variation and marginalization of nuisance parameters leading to somewhat different chi-squared values each time the likelihood is called. I don't know if @fkoehlin is still hanging around here, if so he might be able to make an expert suggestion for how to proceed (sorry for bugging you, Fabian, and I hope all is well!)

Naively, I would think you could fix the nuisance parameters within the code in order to do a chi squared comparison between models, but you would probably want an expert on the likelihood to confirm. In the .data file montepython/likelihoods/kids450_cf_likelihood_public/kids450_cf_likelihood_public.data I see a line with the flag kids450_cf_likelihood_public.marginalize_over_multiplicative_bias_uncertainty = True which might be a good first place to start by turning it off. The next ones I see is the line where you set the nuisance parameters, i.e. the line kids450_cf_likelihood_public.use_nuisance = ['A_bary', 'A_IA'] where you could try to remove one or both.

Note that for an actual parameter inference analysis this is not a good idea, it's just to get stable chi squared valued that you can compare between cosmological models. The chi squared value you get would not be realistic as you're excluding known systematics, but my hope would be that those systematics would be approximately the same across different cosmologies so that turning them off gives you a reliable Delta chi squared value between the models.

Finally, I would recommend you have a look at the references given in the README file montepython/likelihoods/kids450_cf_likelihood_public/README.md to help you make informed decisions on how to go about this and whether what I suggest makes sense. I'm not an expert on this likelihood and it's just my best guess for how to proceed given your goals.

Best, Thejs

[brinckmann]

Ah, I somehow missed that you commented on the bootstrap photoz errors and traced the difference in chi squared to that. That's essentially the kind of thing I was suggesting above. I would suspect that these errors mostly change the uncertainty of the parameter inference rather than the mean value, although if they're highly z dependent they may also change the best fit value. You could try to confirm that, but I don't know how much sense parameter inference makes without that option turned on. so I might be inclined to use the best fit you got with the uncertainty and then turn it off for the chi squared comparison to get stable chi squared values, but again you probably want a second opinion from someone within the KiDS collaboration who worked on this likelihood.

Best, Thejs

[srbPhy]

Dear Thejs,

Thank you for the quick reply and suggestions on how to proceed. I actually did a fresh run with bootstrap_photoz_errors = False and got a very similar bestfit point! So I guess I will keep using my older runs for further analysis. I will go ahead and implement you suggestion, i.e. use the same bestfit point but will turn off the bootstrap_photoz_errors flag while comparing the chi^2 values. Maybe in the future, an expert on this likelihood can suggest an alternative method on handling this.

Cheers, Saurabh

[fkoehlin]

Hi Saurabh and Thejs,

sorry for the late reply but you basically figured it out already. Indeed the scatter in chi^2s is due to how the marginalization over the photo-z errors was implemented back then. It's safe to set bootstrap_photoz_errors=False if you'e interested in the best-fit chi^2 and quite frankly the marginalization over the photo-z errors has only a very negligible impact on the overall errorbars. Due to its bootstrapping nature it prevents the likelihood though from converging exactly to the same chi^2 as found in a previous run.

Depending on your science case it might also be beneficial to look into the KV450-likelihood instead?! There the marginalization over photo-z uncertainties was implemented in a proper Bayesian way (i.e. we're explicilty marginalising over photo-z-nuisance parameters).

Cheers, Fabian

[srbPhy]

Hi Fabian,

Thank you for the confirmation. bootstrap_photoz_errors=False works perfectly for me, chi^2 value does not change on using this. I am also looking into the KV450 likelihood.

Thank you Saurabh