NNPDF / pineappl

PineAPPL is not an extension of APPLgrid
https://nnpdf.github.io/pineappl/
GNU General Public License v3.0
12 stars 3 forks source link

`Grid::optimize` causes `Grid::evolve` to produce wrong `FkTable`s #226

Closed andreab1997 closed 1 year ago

andreab1997 commented 1 year ago

Computing FKs with pineappl==0.6.a5 and comparing their predictions with the predictions that can be obtained from the grids (i.e. using pineko compare) results in some of the FK predictions to be off by up to a 100% factor. Considering all the hadronic FKs in NNPDF40 datasets, most of them are fine but there are a (not negligible) few which are off.

This is the comparison I mentioned
comparison_424.txt

Just skipping the grid optimization in pineko (i.e. grid.optimize()) the problem disappears.

Here is the same comparison but with the new FKs comparison_424.txt

To simplify the debug process, here is the list of the mostly affected dataset:

which of course can be inferred by the comparisons I posted.

@cschwan @AleCandido @felixhekhorn

As far as I know at the moment only me, @niclaurenti and @giacomomagni are computing fktables. I tag you here so that you are aware and you can check your FKs as well.

felixhekhorn commented 1 year ago

Note that for

so the conjecture by @cschwan seems correct there is something fishy with the APPL conversion; however, keep in mind the grids were (most likely) not re-converted, but they were converted long ago

cschwan commented 1 year ago

Good observation. I took @andreab1997's two lists and diff'ed them, which leads to the following two lists; the grids that seem to be unaffected are:

ATLAS_DY_7TEV_49FB_HIMASS.pineappl.lz4
ATLAS_DY_7TEV_CC.pineappl.lz4
ATLAS_DY_7TEV_CF.pineappl.lz4
ATLAS_DY_7TEV_LOMASS_EXT.pineappl.lz4
ATLAS_TTBARTOT_13TEV_FULLLUMI-TOPDIFF13TEVTOT.pineappl.lz4
ATLASTTBARTOT7TEV-TOPDIFF7TEVTOT.pineappl.lz4
ATLASTTBARTOT8TEV-TOPDIFF8TEVTOT.pineappl.lz4
ATLAS_WM_7TEV.pineappl.lz4
ATLAS_WP_7TEV.pineappl.lz4
CMS_DY_7TEV_2D.pineappl.lz4
CMSTTBARTOT13TEV-TOPDIFF13TEVTOT.pineappl.lz4
CMSTTBARTOT7TEV-TOPDIFF7TEVTOT.pineappl.lz4
CMSTTBARTOT8TEV-TOPDIFF8TEVTOT.pineappl.lz4
CMSWEASY840PB-CMS-PAS-SMP-12-001-Wminus_eta3.pineappl.lz4
D0ZRAP-D0_ZNORM_MCgrid_500M.pineappl.lz4
DYE605.pineappl.lz4
DYE866P.pineappl.lz4
DYE866R_D.pineappl.lz4
DYE866R_P.pineappl.lz4
DYE906R_D_bin_0.pineappl.lz4
DYE906R_D_bin_1.pineappl.lz4
DYE906R_D_bin_2.pineappl.lz4
DYE906R_D_bin_3.pineappl.lz4
DYE906R_D_bin_4.pineappl.lz4
DYE906R_D_bin_5.pineappl.lz4
DYE906R_D_bin_6.pineappl.lz4
DYE906R_D_bin_7.pineappl.lz4
DYE906R_D_bin_8.pineappl.lz4
DYE906R_D_bin_9.pineappl.lz4
DYE906R_P_bin_0.pineappl.lz4
DYE906R_P_bin_1.pineappl.lz4
DYE906R_P_bin_2.pineappl.lz4
DYE906R_P_bin_3.pineappl.lz4
DYE906R_P_bin_4.pineappl.lz4
DYE906R_P_bin_5.pineappl.lz4
DYE906R_P_bin_6.pineappl.lz4
DYE906R_P_bin_7.pineappl.lz4
DYE906R_P_bin_8.pineappl.lz4
DYE906R_P_bin_9.pineappl.lz4
LHCB_DY_13TEV_DIELECTRON.pineappl.lz4
LHCB_DY_13TEV_DIMUON.pineappl.lz4
LHCB_DY_7TEV_940PB.pineappl.lz4
LHCB_DY_7TEV.pineappl.lz4
LHCB_DY_8TEV_2FB.pineappl.lz4
LHCB_DY_8TEV.pineappl.lz4
LHCB_WM_7TEV.pineappl.lz4
LHCB_WM_8TEV.pineappl.lz4
LHCB_WP_7TEV.pineappl.lz4
LHCB_WP_8TEV.pineappl.lz4

and the list of clearly affected grids:

ATLASDY2D8TEV-aMCfast_obs_0.pineappl.lz4
ATLASDY2D8TEV-aMCfast_obs_1.pineappl.lz4
ATLASDY2D8TEV-aMCfast_obs_2.pineappl.lz4
ATLASDY2D8TEV-aMCfast_obs_3.pineappl.lz4
ATLASDY2D8TEV-aMCfast_obs_4.pineappl.lz4
ATLAS_DY_2D_8TEV_LOWMASS-ATLAS_DY_2D_8TEV-BIN1_rapidity.pineappl.lz4
ATLAS_DY_2D_8TEV_LOWMASS-ATLAS_DY_2D_8TEV-BIN2_rapidity.pineappl.lz4
ATLAS_DY_2D_8TEV_LOWMASS-ATLAS_DY_2D_8TEV-BIN3_rapidity.pineappl.lz4
ATLAS_DY_2D_8TEV_LOWMASS-ATLAS_DY_2D_8TEV-BIN4_rapidity.pineappl.lz4
ATLAS_DY_2D_8TEV_LOWMASS-ATLAS_DY_2D_8TEV-BIN5_rapidity.pineappl.lz4
ATLAS_DY_2D_8TEV_LOWMASS-ATLAS_DY_2D_8TEV-BIN6_rapidity.pineappl.lz4
ATLAS_DY_2D_8TEV_LOWMASS-ATLAS_DY_2D_8TEV-BIN7_rapidity.pineappl.lz4
ATLASPHT15-ATLASPHT15_Et_1bin.pineappl.lz4
ATLASPHT15-ATLASPHT15_Et_2bin.pineappl.lz4
ATLASPHT15-ATLASPHT15_Et_3bin.pineappl.lz4
ATLASPHT15-ATLASPHT15_Et_4bin.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_TBAR_RAP_NORM-aMCfast_obs_0.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_TBAR_RAP_NORM-aMCfast_obs_1.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_TBAR_RAP_NORM-aMCfast_obs_2.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_TBAR_RAP_NORM-ATLAS_SINGLETOP_TCH_R_7TEV_TB.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_T_RAP_NORM-aMCfast_obs_0.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_T_RAP_NORM-aMCfast_obs_1.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_T_RAP_NORM-aMCfast_obs_2.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_7TEV_T_RAP_NORM-ATLAS_SINGLETOP_TCH_R_7TEV_T.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_TBAR_RAP_NORM-aMCfast_obs_0.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_TBAR_RAP_NORM-aMCfast_obs_1.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_TBAR_RAP_NORM-aMCfast_obs_2.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_TBAR_RAP_NORM-ATLAS_SINGLETOP_TCH_R_8TEV_TB.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_T_RAP_NORM-aMCfast_obs_0.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_T_RAP_NORM-aMCfast_obs_1.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_T_RAP_NORM-aMCfast_obs_2.pineappl.lz4
ATLAS_SINGLETOP_TCH_DIFF_8TEV_T_RAP_NORM-ATLAS_SINGLETOP_TCH_R_8TEV_T.pineappl.lz4
ATLAS_SINGLETOP_TCH_R_13TEV-ATLAS_SINGLETOP_TCH_R_13TEV_TB.pineappl.lz4
ATLAS_SINGLETOP_TCH_R_13TEV-ATLAS_SINGLETOP_TCH_R_13TEV_T.pineappl.lz4
ATLAS_SINGLETOP_TCH_R_7TEV-ATLAS_SINGLETOP_TCH_R_7TEV_TB.pineappl.lz4
ATLAS_SINGLETOP_TCH_R_7TEV-ATLAS_SINGLETOP_TCH_R_7TEV_T.pineappl.lz4
ATLAS_TTB_8TEV_2L_TTRAP.pineappl.lz4
ATLAS_TTB_8TEV_LJ_TRAP.pineappl.lz4
ATLAS_TTB_8TEV_LJ_TRAP_TOT.pineappl.lz4
ATLAS_TTB_8TEV_LJ_TTRAP.pineappl.lz4
ATLAS_TTB_8TEV_LJ_TTRAP_TOT.pineappl.lz4
ATLAS_WM_JET_8TEV_PT-atlas-atlas-wjets-arxiv-1711.03296-xsec003.pineappl.lz4
ATLAS_WP_JET_8TEV_PT-atlas-atlas-wjets-arxiv-1711.03296-xsec002.pineappl.lz4
ATLASWZRAP36PB-ATLAS-arXiv:1109.5141-Wminus_eta3.pineappl.lz4
ATLASWZRAP36PB-ATLAS-arXiv:1109.5141-Wplus_eta4.pineappl.lz4
ATLASWZRAP36PB-ATLAS-arXiv:1109.5141-Z0_eta34.pineappl.lz4
ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_WM_tot.pineappl.lz4
ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_WP_tot.pineappl.lz4
ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_Z_tot.pineappl.lz4
ATLASZPT8TEVMDIST-ATLASZPT8TEV-MLLBIN1_ptZ.pineappl.lz4
ATLASZPT8TEVMDIST-ATLASZPT8TEV-MLLBIN2_ptZ.pineappl.lz4
ATLASZPT8TEVMDIST-ATLASZPT8TEV-MLLBIN3_ptZ.pineappl.lz4
ATLASZPT8TEVMDIST-ATLASZPT8TEV-MLLBIN4_ptZ.pineappl.lz4
ATLASZPT8TEVMDIST-ATLASZPT8TEV-MLLBIN6_ptZ.pineappl.lz4
ATLASZPT8TEVYDIST-ATLASZPT8TEVYDIST-BIN1_ptZ.pineappl.lz4
ATLASZPT8TEVYDIST-ATLASZPT8TEVYDIST-BIN2_ptZ.pineappl.lz4
ATLASZPT8TEVYDIST-ATLASZPT8TEVYDIST-BIN3_ptZ.pineappl.lz4
ATLASZPT8TEVYDIST-ATLASZPT8TEVYDIST-BIN4_ptZ.pineappl.lz4
ATLASZPT8TEVYDIST-ATLASZPT8TEVYDIST-BIN5_ptZ.pineappl.lz4
ATLASZPT8TEVYDIST-ATLASZPT8TEVYDIST-BIN6_ptZ.pineappl.lz4
CDFZRAP_NEW-CDF_ZRAP_MCgrid_500M_last_two_bins_combined.pineappl.lz4
CMS_SINGLETOP_TCH_R_13TEV-CMS_SINGLETOP_TCH_R_13TEV_TB.pineappl.lz4
CMS_SINGLETOP_TCH_R_13TEV-CMS_SINGLETOP_TCH_R_13TEV_T.pineappl.lz4
CMS_SINGLETOP_TCH_R_8TEV-CMS_SINGLETOP_TCH_R_8TEV_TB.pineappl.lz4
CMS_SINGLETOP_TCH_R_8TEV-CMS_SINGLETOP_TCH_R_8TEV_T.pineappl.lz4
CMS_SINGLETOP_TCH_TOT_7TEV-CMS_SINGLETOP_TCH_R_7TEV_TB.pineappl.lz4
CMS_SINGLETOP_TCH_TOT_7TEV-CMS_SINGLETOP_TCH_R_7TEV_T.pineappl.lz4
CMSTOPDIFF8TEVTTRAPNORM-TOPDIFF8TEVTTRAP.pineappl.lz4
CMS_TTB_13TEV_2L_TRAP.pineappl.lz4
CMS_TTB_13TEV_LJ_TRAP.pineappl.lz4
CMS_TTB_5TEV_TOT.pineappl.lz4
CMS_TTB_8TEV_2D_TTM_TRAP.pineappl.lz4
CMS_TTB_8TEV_2D_TTM_TRAP_TOT.pineappl.lz4
CMSWEASY840PB-CMS-PAS-SMP-12-001-Wminus_eta3.pineappl.lz4
CMSWEASY840PB-CMS-PAS-SMP-12-001-Wplus_eta4.pineappl.lz4
CMSWMASY47FB-CMSWMASY47FB-Wm-eta3.pineappl.lz4
CMSWMASY47FB-CMSWMASY47FB-Wp-eta4.pineappl.lz4
CMSWMU8TEV-CMSWMU8TEV_WM_leptrap.pineappl.lz4
CMSWMU8TEV-CMSWMU8TEV_WP_leptrap.pineappl.lz4
CMSZDIFF12-CMSZDIFF12-BIN1_ptZ.pineappl.lz4
CMSZDIFF12-CMSZDIFF12-BIN2_ptZ.pineappl.lz4
CMSZDIFF12-CMSZDIFF12-BIN3_ptZ.pineappl.lz4
CMSZDIFF12-CMSZDIFF12-BIN4_ptZ.pineappl.lz4
CMSZDIFF12-CMSZDIFF12-BIN5_ptZ.pineappl.lz4
D0WMASY-grid-40-6-15-3-Wminus_wly_pt25.pineappl.lz4
D0WMASY-grid-40-6-15-3-Wplus_wly_pt25.pineappl.lz4
D0ZRAP-D0_ZRAP_MCgrid_500M.pineappl.lz4
felixhekhorn commented 1 year ago

@andreab1997 can you check whether this is a NNLO problem? i.e can you compute the NLO EKO counterpart from the grid and compare grid vs FK at NLO?

using an old version of ATLASWZRAP36PB-ATLAS-arXiv:1109.5141-Z0_eta34.pineappl.lz4 which I had floating around somewhere I got

$ pineappl evolve too.pineappl.lz4 t.tar tee.pineappl.lz4 NNPDF40_nlo_as_01180
LHAPDF 6.4.0 loading /home/felix/local/share/LHAPDF/NNPDF40_nlo_as_01180/NNPDF40_nlo_as_01180_0000.dat
NNPDF40_nlo_as_01180 PDF set, member #0, version 1; LHAPDF ID = 331700
b    Grid       FkTable     rel. diff
-+-----------+-----------+-------------
0 1.3463245e5 1.3450064e5 -9.7904355e-4
1 1.3347254e5 1.3334423e5 -9.6133745e-4

with everything at NLO (note that the EKO was also an old eko so I could run pineappl evolve)

too.pineappl.lz4 is a re-optimized version of the original grid and that seems to be more or less fine:

$ pineappl diff t.pineappl.lz4 too.pineappl.lz4 NNPDF40_nlo_as_01180 --ignore-lumis
LHAPDF 6.4.0 loading /home/felix/local/share/LHAPDF/NNPDF40_nlo_as_01180/NNPDF40_nlo_as_01180_0000.dat
NNPDF40_nlo_as_01180 PDF set, member #0, version 1; LHAPDF ID = 331700
b   x1               O(as^0 a^0)                        O(as^1 a^0)            
-+---+---+-----------+-----------+----------+-----------+-----------+----------
0   0 0.4 1.2425259e5 1.2425259e5  2.220e-16 1.0379858e4 1.0379858e4  8.882e-16
1 0.4 0.8 1.2306826e5 1.2306826e5  2.220e-16 1.0404277e4 1.0404277e4 -4.774e-15

Note the additional --ignore-lumis though

cschwan commented 1 year ago

CMS_TTB_8TEV_2D_TTM_TRAP_TOT.pineappl.lz4 shows the largest differences and this is a grid that we generated ourselves, so it's mostly the converted grids but not only them.

andreab1997 commented 1 year ago

@andreab1997 can you check whether this is a NNLO problem? i.e can you compute the NLO EKO counterpart from the grid and compare grid vs FK at NLO?

using an old version of ATLASWZRAP36PB-ATLAS-arXiv:1109.5141-Z0_eta34.pineappl.lz4 which I had floating around somewhere I got

$ pineappl evolve too.pineappl.lz4 t.tar tee.pineappl.lz4 NNPDF40_nlo_as_01180
LHAPDF 6.4.0 loading /home/felix/local/share/LHAPDF/NNPDF40_nlo_as_01180/NNPDF40_nlo_as_01180_0000.dat
NNPDF40_nlo_as_01180 PDF set, member #0, version 1; LHAPDF ID = 331700
b    Grid       FkTable     rel. diff
-+-----------+-----------+-------------
0 1.3463245e5 1.3450064e5 -9.7904355e-4
1 1.3347254e5 1.3334423e5 -9.6133745e-4

with everything at NLO (note that the EKO was also an old eko so I could run pineappl evolve)

too.pineappl.lz4 is a re-optimized version of the original grid and that seems to be more or less fine:

$ pineappl diff t.pineappl.lz4 too.pineappl.lz4 NNPDF40_nlo_as_01180 --ignore-lumis
LHAPDF 6.4.0 loading /home/felix/local/share/LHAPDF/NNPDF40_nlo_as_01180/NNPDF40_nlo_as_01180_0000.dat
NNPDF40_nlo_as_01180 PDF set, member #0, version 1; LHAPDF ID = 331700
b   x1               O(as^0 a^0)                        O(as^1 a^0)            
-+---+---+-----------+-----------+----------+-----------+-----------+----------
0   0 0.4 1.2425259e5 1.2425259e5  2.220e-16 1.0379858e4 1.0379858e4  8.882e-16
1 0.4 0.8 1.2306826e5 1.2306826e5  2.220e-16 1.0404277e4 1.0404277e4 -4.774e-15

Note the additional --ignore-lumis though

I already did and yes, NLO seems to be fine. Let me post here the usual comparison. comparison_405.txt

cschwan commented 1 year ago

Does that mean there's some interference with the NNLO K-factor to grid conversion and evolution? That's strange.

andreab1997 commented 1 year ago

Does that mean there's some interference with the NNLO K-factor to grid conversion and evolution? That's strange.

I don't believe so because the same problem that I had with the grids with included kfactors, I also had with the usual grids once I used them at NNLO (which is what we usually do in order to use the kfactor at the end)

cschwan commented 1 year ago

To effectively debug this, I need to upgrade pineappl evolve ... for which I've opened #227.

cschwan commented 1 year ago

This isn't a bug in PineAPPL.