Plotting shifted data in validphys

[juanrojochacon]

So far in validphys (as well as in most NNPDF) papers when we compare data and theory we plot the actual central value of the experimental data and then add in quadrature systematic and statistical errors in the total error bar. However in many cases, in particular when systematics dominate, it can be quite misleading to judge the overal agreement between data and theory.

A solution for this is to shift the experimental data (or the theory, is equivalent) according to the best-fit values of their systematic uncertainties. This is something that was done for example in the NNPDF3.1sx paper, see eg Fig. 5.1 in https://arxiv.org/pdf/1710.05935.pdf. In this case, in the right plots the theory has been shifted and then only the total uncorrelated error is shown together with the experimental data.

It would be very nice that in validphys one could plot data this way. The analytic expressions for these shifts can be found in many places, for example in https://arxiv.org/pdf/1709.04922.pdf in Sect 4.2.1. in particular using Eq, (85). By construction the actual value of the chi2 will be very similar but now it will be clear to which extent the systematic errors can account for the agreement or lack thereof between the theory and data.

I hope that what I meant is now clearer @Zaharid . It is not a super urgent feature but I think in general is good practice and would be very neat to be able to produce these kind of plots for the NNPDF4.0 paper.

[Zaharid]

One thing I don't understand is that I would say that if anything the plots are misleading in the other direction. Namely they might look good because the prediction is within the error bars, but result in a bad chi² because some shift was in a direction with high correlations. Wouldn't we make the plots even more misleading in this way?

[juanrojochacon]

I don't think so: the chi2 is unchanged, and with the shifted comparison one sees how well data and theory agree within the uncorrelated (point by point) error. So it is a bit cleaner in the sense that it allows the reader to see if the bad chi2 arises due to the correlated or the uncorrelated component of the experimental error

[Zaharid]

I don't think so: the chi2 is unchanged, and with the shifted comparison one sees how well data and theory agree within the uncorrelated (point by point) error.

I am confused now: Clearly if we had no uncorrelated error there would be no shifts in the suggested procedure, right? And so we would have the same plots as now. Note however that nothing stops us from shifting according to e.g. the uncorrelated statistical error.

[juanrojochacon]

not sure I follow. The shift involves only systematic correlated errors: if one has only statistical errors there is nothing to shift, as indicated in the analytic Hessian formulae. In the limit where there are no uncorrelated errors the chi2 is not well defined since one is dividing by zero

[Zaharid]

I guess what I am saying is that I don't see in which sense plots constructed in this way give a less misleading description of reality (and I am probably missing something). You say:

I don't think so: the chi2 is unchanged, and with the shifted comparison one sees how well data and theory agree within the uncorrelated (point by point) error. So it is a bit cleaner in the sense that it allows the reader to see if the bad chi2 arises due to the correlated or the uncorrelated component of the experimental error

But the thing is, if there are no correlated errors we are both saying that we have nothing to do. If there are correlated errors and we have good looking plots as done currently and a bad chi², then the chi² comes from the correlations, so it looks to me it is also fine for the stated purpose. It looks to me that what you are suggesting is more in line of removing the correlated systematics from the comparison altogether, but then we could as well do that and gain some clarity in the process? I'd also note that if I understand the logic in the text (which I am not sure), then we should shift the central value but keep the original error bar in place (because the distribution of the nuisance is fixed and we are shifting according to the best fit value from that distribution).

PS: One thing that is confusing from the text in https://arxiv.org/pdf/1709.04922.pdf is that it calls uncorrelated error what in reality is variance (i.e. s_k in eq 85).

[Zaharid]

btw, one thing that would achieve perhaps a similar purpose is to plot the chi² contribution per data point, ie

contribution of point j is:

(sum_i covmat^(-1/2)_{ij} delta_j)**2

and then the total chi² is the sum of such contributions.

[juanrojochacon]

Sorry I forgot to reply to this @Zaharid . I agree with what you say, it is just a matter of visualisation. The point of the shifted data is to display only the uncorrelated uncertainties, and include the correlated ones in the shifts. This does not affect the chi2 by any means, but allows to gauge better the agreement between theory and data. In the presence of large systematic errors, visually in the traditional way one can seem to have good agreement but the chi2 is poor, and this has been the source of endless confusion

[Zaharid]

Sorry but I am not sure I understand this.

In the presence of large systematic errors, visually in the traditional way one can seem to have good agreement but the chi2 is poor, and this has been the source of endless confusion

As I said in the first comment here and as shown in the explicit example by Rabah in the PR, these shifts increase the visual agreement when the dataset disagree. Hence the last few comments here...

[juanrojochacon]

the chi2 is identical in the shifted and unshifted data, so it is just a matter of the visualisation.... I can reply later once I am done with the PDF4LHC benchmarking meeting ;)