search

NNPDF / nnpdf

An open-source machine learning framework for global analyses of parton distributions.
https://docs.nnpdf.science/
GNU General Public License v3.0
30 stars 6 forks source link

CMSZDIFF12 chi2 #345

Closed scarrazza closed 5 years ago

scarrazza commented 5 years ago

Following our discussing today, I found the commit which deteriorates the chi2 of CMSZDIFF12 from 1.32663 to 3.53007 when using NNPDF3.1 NNLO.

https://github.com/NNPDF/nnpdf/commit/97c4c5b3a302f8921ff5430ffd69616b1c398bdd

As you see, this derives from the fix of the cholesky decomposition, not sure exacly how this affects only the zdiff computation.

scarrazza commented 5 years ago

@tgiani @Zaharid could you please have a look?

tgiani commented 5 years ago

Actually I think CMSZDIFF12 was also one of the dataset where we have changed the treatment of the systematics

scarrazza commented 5 years ago

@tgiani the treatment of systematics has been changed after this commit, in particular the 12th november in fbce60f5816cfde3e0166b1b5513c462d1868aba.

If you try commit 082b387825d034934dce2b013f4f4ee010b5bea5 (one before your cholesky fix) the chi2 are 1.32663 as in the NNPDF3.1 paper.

Zaharid commented 5 years ago

Stefano did you change the data files?

On Fri, 7 Dec 2018, 13:07 Stefano Carrazza <notifications@github.com wrote:

@tgiani https://github.com/tgiani the treatment of systematics have been changed after this commit, in particular the 12th november in fbce60f https://github.com/NNPDF/nnpdf/commit/fbce60f5816cfde3e0166b1b5513c462d1868aba .

If you try commit 082b387 https://github.com/NNPDF/nnpdf/commit/082b387825d034934dce2b013f4f4ee010b5bea5 (one before your cholesky fix) the chi2 are 1.32663 as in the NNPDF3.1 paper.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NNPDF/nnpdf/issues/345#issuecomment-445227340, or mute the thread https://github.com/notifications/unsubscribe-auth/AFabUs_y3B8UseXzizbCHu3fDQOcmw1Aks5u2mf5gaJpZM4ZIYry .

tgiani commented 5 years ago

A very stupid test: I print the cov mat for CMSZDIFF12, then I compute its sqrt using cholesky and I print it, and finally I check that taking the sqrt times its transpose I get again the initial cov mat

-- Generating replica data for CMS
Covariance matrix
===============================================
6959 2480 1057 508.8 251.4 140 68.27 6590 2402 1021 491.6 251.4 130.5 67.31 5972 2151 952.8 457.9 235.4 121.4 64.18 4458 1640 723.6 364.5 192.8 106.7 54.23 
2480 1003 418.7 206.9 103.4 57.47 28.03 2495 939.1 405.7 198.7 102.5 53.75 27.58 2229 822.9 368.6 179.9 93.13 48.44 25.41 1658 634.8 282.4 143.1 75.52 42.17 21.44 
1057 418.7 191.6 90.68 45.51 25.47 12.45 1069 406.9 177.3 87.11 45.09 23.85 12.2 955.6 355.6 160.7 78.35 40.97 21.43 11.22 710.5 274.2 123.3 62.32 32.92 18.54 9.444 
508.8 206.9 90.68 49.75 22.73 12.73 6.219 521.1 200.8 87.83 43.66 22.62 11.94 6.099 465 175.6 79.2 39.27 20.36 10.61 5.563 345 135.3 60.59 31.07 16.34 9.162 4.664 
251.4 103.4 45.51 22.73 13.3 6.276 3.121 259 100.2 44.05 21.85 11.37 5.98 3.081 231 87.46 39.52 19.52 10.22 5.328 2.788 171.4 67.51 30.26 15.6 8.156 4.551 2.353 
140 57.47 25.47 12.73 6.276 4.595 1.731 144.6 55.85 24.62 12.13 6.318 3.358 1.72 128.9 48.52 22.17 10.86 5.655 2.999 1.59 95.39 37.53 17.08 8.588 4.548 2.631 1.318 
68.27 28.03 12.45 6.219 3.121 1.731 1.168 70.26 27.25 12.05 5.938 3.081 1.661 0.8452 62.72 23.62 10.78 5.295 2.762 1.469 0.7648 46.41 18.23 8.264 4.202 2.203 1.251 0.6424 
6590 2495 1069 521.1 259 144.6 70.26 6724 2442 1045 506.7 259.8 136 70.12 5962 2180 969.5 468.8 241.4 125.7 66.32 4449 1684 748 377.2 199.1 111.4 56.4 
2402 939.1 406.9 200.8 100.2 55.85 27.25 2442 955.6 397.7 195.4 100.7 52.72 27.11 2205 821.8 368.2 179.7 93.18 48.48 25.5 1646 635.7 283.5 144.3 75.85 42.39 21.59 
1021 405.7 177.3 87.83 44.05 24.62 12.05 1045 397.7 181.8 84.76 44.06 23.25 11.98 941.9 353.1 159.7 78 40.51 21.22 11.2 702.1 272.7 122.7 62.26 32.88 18.46 9.445 
491.6 198.7 87.11 43.66 21.85 12.13 5.938 506.7 195.4 84.76 46.39 21.74 11.49 5.917 455.7 172.8 78.12 38.5 20.08 10.43 5.486 339.2 133.3 59.88 30.69 16.2 9.012 4.623 
251.4 102.5 45.09 22.62 11.37 6.318 3.081 259.8 100.7 44.06 21.74 13.14 5.849 3.059 233.5 88.77 40.24 19.85 10.44 5.424 2.821 173.7 68.74 30.85 15.82 8.378 4.667 2.39 
130.5 53.75 23.85 11.94 5.98 3.358 1.661 136 52.72 23.25 11.49 5.849 4.109 1.581 121.9 46.38 21.13 10.39 5.425 2.896 1.503 90.87 35.88 16.32 8.286 4.383 2.501 1.274 
67.31 27.58 12.2 6.099 3.081 1.72 0.8452 70.12 27.11 11.98 5.917 3.059 1.581 1.14 62.93 23.84 10.85 5.319 2.804 1.493 0.7758 46.88 18.44 8.41 4.253 2.256 1.269 0.667 
5972 2229 955.6 465 231 128.9 62.72 5962 2205 941.9 455.7 233.5 121.9 62.93 5546 1997 889.4 429.4 221 115 60.94 4074 1541 685.8 347 183 101.9 51.81 
2151 822.9 355.6 175.6 87.46 48.52 23.62 2180 821.8 353.1 172.8 88.77 46.38 23.84 1997 781.2 336.1 164.6 84.78 44.22 23.39 1506 580.6 260 132.8 70.08 38.88 19.86 
952.8 368.6 160.7 79.2 39.52 22.17 10.78 969.5 368.2 159.7 78.12 40.24 21.13 10.85 889.4 336.1 160.6 74.23 38.74 20.28 10.7 671.2 261.2 118 60.3 31.91 17.82 9.072 
457.9 179.9 78.35 39.27 19.52 10.86 5.295 468.8 179.7 78 38.5 19.85 10.39 5.319 429.4 164.6 74.23 41 18.95 10.07 5.277 325.7 128.4 58.08 29.83 15.84 8.751 4.487 
235.4 93.13 40.97 20.36 10.22 5.655 2.762 241.4 93.18 40.51 20.08 10.44 5.425 2.804 221 84.78 38.74 18.95 11.81 5.068 2.738 167.5 66.41 30.05 15.48 8.163 4.535 2.318 
121.4 48.44 21.43 10.61 5.328 2.999 1.469 125.7 48.48 21.22 10.43 5.424 2.896 1.493 115 44.22 20.28 10.07 5.068 3.736 1.404 87.74 34.86 15.95 8.135 4.351 2.444 1.254 
64.18 25.41 11.22 5.563 2.788 1.59 0.7648 66.32 25.5 11.2 5.486 2.821 1.503 0.7758 60.94 23.39 10.7 5.277 2.738 1.404 1.082 46.21 18.34 8.423 4.29 2.272 1.286 0.6527 
4458 1658 710.5 345 171.4 95.39 46.41 4449 1646 702.1 339.2 173.7 90.87 46.88 4074 1506 671.2 325.7 167.5 87.74 46.21 3173 1173 525.2 265.9 140.9 77.98 39.54 
1640 634.8 274.2 135.3 67.51 37.53 18.23 1684 635.7 272.7 133.3 68.74 35.88 18.44 1541 580.6 261.2 128.4 66.41 34.86 18.34 1173 486.5 206.6 106 56.1 31.08 15.74 
723.6 282.4 123.3 60.59 30.26 17.08 8.264 748 283.5 122.7 59.88 30.85 16.32 8.41 685.8 260 118 58.08 30.05 15.95 8.423 525.2 206.6 101.3 47.83 25.72 14.22 7.331 
364.5 143.1 62.32 31.07 15.6 8.588 4.202 377.2 144.3 62.26 30.69 15.82 8.286 4.253 347 132.8 60.3 29.83 15.48 8.135 4.29 265.9 106 47.83 28.38 12.86 7.345 3.694 
192.8 75.52 32.92 16.34 8.156 4.548 2.203 199.1 75.85 32.88 16.2 8.378 4.383 2.256 183 70.08 31.91 15.84 8.163 4.351 2.272 140.9 56.1 25.72 12.86 8.611 3.701 1.984 
106.7 42.17 18.54 9.162 4.551 2.631 1.251 111.4 42.39 18.46 9.012 4.667 2.501 1.269 101.9 38.88 17.82 8.751 4.535 2.444 1.286 77.98 31.08 14.22 7.345 3.701 3.088 1.032 
54.23 21.44 9.444 4.664 2.353 1.318 0.6424 56.4 21.59 9.445 4.623 2.39 1.274 0.667 51.81 19.86 9.072 4.487 2.318 1.254 0.6527 39.54 15.74 7.331 3.694 1.984 1.032 0.8581 

Sqrt of Covariance Matrix using Cholesky
===============================================
83.4206 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
29.7289 10.9176 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
12.6707 3.84824 4.03035 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
6.09921 2.34276 1.08751 2.42454 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
3.01364 1.26473 0.609867 0.298187 1.46887 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
1.67824 0.694085 0.380722 0.187223 0.0357634 1.05616 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0.818383 0.338939 0.192583 0.0923975 0.055158 0.028131 0.57785 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
78.9973 13.4188 4.07124 1.40831 0.719257 0.824009 0.146794 16.8402 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
28.7938 7.6108 3.16918 1.60999 0.944406 0.665279 0.47785 2.89607 6.78268 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
12.2392 3.83262 1.8539 0.901456 0.62554 0.494779 0.425542 1.00782 0.700176 3.27732 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
5.89303 2.15312 1.03101 0.639996 0.372935 0.208241 0.175398 0.39847 0.487085 0.22585 2.21212 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
3.01364 1.18229 0.584369 0.343909 0.227212 0.137073 0.0921652 0.161054 0.251459 0.202386 0.11324 1.4066 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
1.56436 0.663451 0.366044 0.184054 0.101009 0.0896633 0.104336 0.0953503 0.101613 0.121145 0.0714273 -0.0322713 0.992158 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0.806875 0.329057 0.176168 0.0887644 0.067598 0.0486362 0.0451881 0.0609248 0.0556029 0.0765372 0.0466588 0.0242384 -0.0197198 0.562999 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
71.589 9.22729 3.22708 1.33496 0.83064 0.798661 0.332587 9.88839 4.56483 2.72791 1.11241 0.536507 0.373493 0.374241 13.9417 0 0 0 0 0 0 0 0 0 0 0 0 0 
25.785 5.1606 2.23955 1.56991 0.948034 0.458426 0.220363 3.64524 2.7404 1.72733 1.24681 0.744895 0.453822 0.235228 2.69684 6.90217 0 0 0 0 0 0 0 0 0 0 0 0 
11.4216 2.66063 1.4244 0.723751 0.442376 0.436869 0.264881 1.42423 1.24415 1.06983 0.648253 0.40906 0.308636 0.192659 1.22006 1.1331 3.4416 0 0 0 0 0 0 0 0 0 0 0 
5.48905 1.53117 0.721381 0.585447 0.290651 0.180494 0.120711 0.62331 0.664865 0.5092 0.399286 0.253823 0.149501 0.0666034 0.540235 0.730574 0.465852 2.26891 0 0 0 0 0 0 0 0 0 0 
2.82184 0.846317 0.4859 0.263074 0.184379 0.086166 0.06113 0.271029 0.365355 0.268032 0.232952 0.202951 0.0852578 0.0867155 0.251397 0.331989 0.335023 0.148985 1.44347 0 0 0 0 0 0 0 0 0 
1.45528 0.474124 0.289319 0.127269 0.0873364 0.0856987 0.0737615 0.170643 0.173652 0.158179 0.0912129 0.0803698 0.0950809 0.0898807 0.146969 0.182191 0.187224 0.151559 -0.0208982 1.02158 0 0 0 0 0 0 0 0 
0.769354 0.232466 0.143198 0.0702009 0.0457296 0.0645632 0.0311146 0.0980512 0.0923344 0.0997055 0.0526919 0.023114 0.0326584 0.0325463 0.0934966 0.101652 0.0965679 0.0652021 0.0397605 -0.00846689 0.579285 0 0 0 0 0 0 0 
53.44 6.34652 2.22141 0.731611 0.511483 0.282787 -0.0122002 7.81158 4.04682 2.37252 0.897766 0.416814 0.546494 0.434149 5.53378 4.08501 2.56218 1.72251 1.01678 1.23608 0.728859 11.2537 0 0 0 0 0 0 
19.6594 4.61164 1.82473 1.07432 0.679372 0.393627 0.136355 3.52152 2.33716 1.28024 0.794836 0.626068 0.319757 0.148015 2.32572 2.01291 1.50874 1.24837 0.863436 0.741888 0.531535 1.53972 5.97717 0 0 0 0 0 
8.67411 2.24668 1.17779 0.470378 0.285474 0.394508 0.184544 1.57991 1.02529 0.779696 0.39982 0.25008 0.291341 0.255438 1.13632 1.01682 0.925499 0.701114 0.441459 0.477518 0.433489 0.864683 0.645868 3.02586 0 0 0 0 
4.36942 1.20924 0.571362 0.39828 0.296518 0.107023 0.0866673 0.748196 0.629959 0.384024 0.295407 0.193002 0.121541 0.0679778 0.608139 0.622938 0.535985 0.409645 0.287343 0.223443 0.190073 0.403382 0.446644 0.264957 2.04252 0 0 0 
2.31118 0.623878 0.306387 0.185121 0.108818 0.0767481 0.0274179 0.385855 0.294963 0.23006 0.177094 0.13644 0.0843925 0.0634422 0.306749 0.337626 0.309441 0.259104 0.138667 0.162781 0.106986 0.245149 0.243798 0.26596 0.00767653 1.3286 0 0 
1.27906 0.37966 0.216446 0.097297 0.0375708 0.11262 0.0339936 0.24466 0.146494 0.127679 0.0589812 0.0481213 0.0778944 0.0364158 0.164887 0.158809 0.17723 0.109297 0.0676893 0.101171 0.0862944 0.108069 0.114122 0.110392 0.109825 -0.0676182 0.976697 0 
0.650079 0.193622 0.114611 0.0498158 0.0437503 0.0360682 0.0253661 0.1096 0.08254 0.0808119 0.0428207 0.0291737 0.036705 0.0512455 0.0859122 0.086814 0.08177 0.0627653 0.0318934 0.0539456 0.0356319 0.0444345 0.045463 0.0891207 0.0284362 0.0338017 -0.0489037 0.547222 

Sqrt * Sqrt
===============================================
6959 2480 1057 508.8 251.4 140 68.27 6590 2402 1021 491.6 251.4 130.5 67.31 5972 2151 952.8 457.9 235.4 121.4 64.18 4458 1640 723.6 364.5 192.8 106.7 54.23 
2480 1003 418.7 206.9 103.4 57.47 28.03 2495 939.1 405.7 198.7 102.5 53.75 27.58 2229 822.9 368.6 179.9 93.13 48.44 25.41 1658 634.8 282.4 143.1 75.52 42.17 21.44 
1057 418.7 191.6 90.68 45.51 25.47 12.45 1069 406.9 177.3 87.11 45.09 23.85 12.2 955.6 355.6 160.7 78.35 40.97 21.43 11.22 710.5 274.2 123.3 62.32 32.92 18.54 9.444 
508.8 206.9 90.68 49.75 22.73 12.73 6.219 521.1 200.8 87.83 43.66 22.62 11.94 6.099 465 175.6 79.2 39.27 20.36 10.61 5.563 345 135.3 60.59 31.07 16.34 9.162 4.664 
251.4 103.4 45.51 22.73 13.3 6.276 3.121 259 100.2 44.05 21.85 11.37 5.98 3.081 231 87.46 39.52 19.52 10.22 5.328 2.788 171.4 67.51 30.26 15.6 8.156 4.551 2.353 
140 57.47 25.47 12.73 6.276 4.595 1.731 144.6 55.85 24.62 12.13 6.318 3.358 1.72 128.9 48.52 22.17 10.86 5.655 2.999 1.59 95.39 37.53 17.08 8.588 4.548 2.631 1.318 
68.27 28.03 12.45 6.219 3.121 1.731 1.168 70.26 27.25 12.05 5.938 3.081 1.661 0.8452 62.72 23.62 10.78 5.295 2.762 1.469 0.7648 46.41 18.23 8.264 4.202 2.203 1.251 0.6424 
6590 2495 1069 521.1 259 144.6 70.26 6724 2442 1045 506.7 259.8 136 70.12 5962 2180 969.5 468.8 241.4 125.7 66.32 4449 1684 748 377.2 199.1 111.4 56.4 
2402 939.1 406.9 200.8 100.2 55.85 27.25 2442 955.6 397.7 195.4 100.7 52.72 27.11 2205 821.8 368.2 179.7 93.18 48.48 25.5 1646 635.7 283.5 144.3 75.85 42.39 21.59 
1021 405.7 177.3 87.83 44.05 24.62 12.05 1045 397.7 181.8 84.76 44.06 23.25 11.98 941.9 353.1 159.7 78 40.51 21.22 11.2 702.1 272.7 122.7 62.26 32.88 18.46 9.445 
491.6 198.7 87.11 43.66 21.85 12.13 5.938 506.7 195.4 84.76 46.39 21.74 11.49 5.917 455.7 172.8 78.12 38.5 20.08 10.43 5.486 339.2 133.3 59.88 30.69 16.2 9.012 4.623 
251.4 102.5 45.09 22.62 11.37 6.318 3.081 259.8 100.7 44.06 21.74 13.14 5.849 3.059 233.5 88.77 40.24 19.85 10.44 5.424 2.821 173.7 68.74 30.85 15.82 8.378 4.667 2.39 
130.5 53.75 23.85 11.94 5.98 3.358 1.661 136 52.72 23.25 11.49 5.849 4.109 1.581 121.9 46.38 21.13 10.39 5.425 2.896 1.503 90.87 35.88 16.32 8.286 4.383 2.501 1.274 
67.31 27.58 12.2 6.099 3.081 1.72 0.8452 70.12 27.11 11.98 5.917 3.059 1.581 1.14 62.93 23.84 10.85 5.319 2.804 1.493 0.7758 46.88 18.44 8.41 4.253 2.256 1.269 0.667 
5972 2229 955.6 465 231 128.9 62.72 5962 2205 941.9 455.7 233.5 121.9 62.93 5546 1997 889.4 429.4 221 115 60.94 4074 1541 685.8 347 183 101.9 51.81 
2151 822.9 355.6 175.6 87.46 48.52 23.62 2180 821.8 353.1 172.8 88.77 46.38 23.84 1997 781.2 336.1 164.6 84.78 44.22 23.39 1506 580.6 260 132.8 70.08 38.88 19.86 
952.8 368.6 160.7 79.2 39.52 22.17 10.78 969.5 368.2 159.7 78.12 40.24 21.13 10.85 889.4 336.1 160.6 74.23 38.74 20.28 10.7 671.2 261.2 118 60.3 31.91 17.82 9.072 
457.9 179.9 78.35 39.27 19.52 10.86 5.295 468.8 179.7 78 38.5 19.85 10.39 5.319 429.4 164.6 74.23 41 18.95 10.07 5.277 325.7 128.4 58.08 29.83 15.84 8.751 4.487 
235.4 93.13 40.97 20.36 10.22 5.655 2.762 241.4 93.18 40.51 20.08 10.44 5.425 2.804 221 84.78 38.74 18.95 11.81 5.068 2.738 167.5 66.41 30.05 15.48 8.163 4.535 2.318 
121.4 48.44 21.43 10.61 5.328 2.999 1.469 125.7 48.48 21.22 10.43 5.424 2.896 1.493 115 44.22 20.28 10.07 5.068 3.736 1.404 87.74 34.86 15.95 8.135 4.351 2.444 1.254 
64.18 25.41 11.22 5.563 2.788 1.59 0.7648 66.32 25.5 11.2 5.486 2.821 1.503 0.7758 60.94 23.39 10.7 5.277 2.738 1.404 1.082 46.21 18.34 8.423 4.29 2.272 1.286 0.6527 
4458 1658 710.5 345 171.4 95.39 46.41 4449 1646 702.1 339.2 173.7 90.87 46.88 4074 1506 671.2 325.7 167.5 87.74 46.21 3173 1173 525.2 265.9 140.9 77.98 39.54 
1640 634.8 274.2 135.3 67.51 37.53 18.23 1684 635.7 272.7 133.3 68.74 35.88 18.44 1541 580.6 261.2 128.4 66.41 34.86 18.34 1173 486.5 206.6 106 56.1 31.08 15.74 
723.6 282.4 123.3 60.59 30.26 17.08 8.264 748 283.5 122.7 59.88 30.85 16.32 8.41 685.8 260 118 58.08 30.05 15.95 8.423 525.2 206.6 101.3 47.83 25.72 14.22 7.331 
364.5 143.1 62.32 31.07 15.6 8.588 4.202 377.2 144.3 62.26 30.69 15.82 8.286 4.253 347 132.8 60.3 29.83 15.48 8.135 4.29 265.9 106 47.83 28.38 12.86 7.345 3.694 
192.8 75.52 32.92 16.34 8.156 4.548 2.203 199.1 75.85 32.88 16.2 8.378 4.383 2.256 183 70.08 31.91 15.84 8.163 4.351 2.272 140.9 56.1 25.72 12.86 8.611 3.701 1.984 
106.7 42.17 18.54 9.162 4.551 2.631 1.251 111.4 42.39 18.46 9.012 4.667 2.501 1.269 101.9 38.88 17.82 8.751 4.535 2.444 1.286 77.98 31.08 14.22 7.345 3.701 3.088 1.032 
54.23 21.44 9.444 4.664 2.353 1.318 0.6424 56.4 21.59 9.445 4.623 2.39 1.274 0.667 51.81 19.86 9.072 4.487 2.318 1.254 0.6527 39.54 15.74 7.331 3.694 1.984 1.032 0.8581

at a first look I would say that it s working

Zaharid commented 5 years ago

@scarrazza I tried the same test and cannot reproduce. I get the same bad chi² for both. I may have forgotten to recompile or something, but I don't think so. Also that change shouldn't make any difference...

I am doing

chi2check 181023-001-sc

After change:

Values of chi2 by dataset
-------------------------- 

Experiment:              NMC    Npts:    325    chi2(cent|diag):     1.29935  |  1.00332
Dataset:            NMCPD   Npts:    121    chi2(cent|diag):     0.92692  |  0.90968
Dataset:              NMC   Npts:    204    chi2(cent|diag):     1.52025  |  1.05885

Experiment:             SLAC    Npts:    67 chi2(cent|diag):     0.73120  |  0.68675
Dataset:            SLACP   Npts:    33 chi2(cent|diag):     0.78722  |  0.80192
Dataset:            SLACD   Npts:    34 chi2(cent|diag):     0.68787  |  0.57496

Experiment:            BCDMS    Npts:    581    chi2(cent|diag):     1.20481  |  0.58762
Dataset:           BCDMSP   Npts:    333    chi2(cent|diag):     1.28552  |  0.67808
Dataset:           BCDMSD   Npts:    248    chi2(cent|diag):     1.09679  |  0.46616

Experiment:           CHORUS    Npts:    832    chi2(cent|diag):     1.12325  |  1.09525
Dataset:         CHORUSNU   Npts:    416    chi2(cent|diag):     1.14197  |  1.30722
Dataset:         CHORUSNB   Npts:    416    chi2(cent|diag):     1.05992  |  0.88329

Experiment:           NTVDMN    Npts:    76 chi2(cent|diag):     0.84904  |  0.87437
Dataset:         NTVNUDMN   Npts:    39 chi2(cent|diag):     0.64963  |  0.70954
Dataset:         NTVNBDMN   Npts:    37 chi2(cent|diag):     1.05604  |  1.04811

Experiment:         HERACOMB    Npts:    1145   chi2(cent|diag):     1.15963  |  2.27524
Dataset:     HERACOMBNCEM   Npts:    159    chi2(cent|diag):     1.41558  |  2.43291
Dataset:  HERACOMBNCEP460   Npts:    204    chi2(cent|diag):     1.07406  |  1.97182
Dataset:  HERACOMBNCEP575   Npts:    254    chi2(cent|diag):     0.89981  |  1.49385
Dataset:  HERACOMBNCEP820   Npts:    70 chi2(cent|diag):     1.14477  |  1.55973
Dataset:  HERACOMBNCEP920   Npts:    377    chi2(cent|diag):     1.30151  |  3.26532
Dataset:     HERACOMBCCEM   Npts:    42 chi2(cent|diag):     1.15985  |  1.26672
Dataset:     HERACOMBCCEP   Npts:    39 chi2(cent|diag):     1.13053  |  1.10825

Experiment:      HERAF2CHARM    Npts:    37 chi2(cent|diag):     1.49044  |  0.98288
Dataset:      HERAF2CHARM   Npts:    37 chi2(cent|diag):     1.49044  |  0.98288

Experiment:         F2BOTTOM    Npts:    29 chi2(cent|diag):     1.10977  |  1.17429
Dataset:        H1HERAF2B   Npts:    12 chi2(cent|diag):     0.77584  |  0.43934
Dataset:      ZEUSHERAF2B   Npts:    17 chi2(cent|diag):     1.34549  |  1.69308

Experiment:           DYE886    Npts:    104    chi2(cent|diag):     1.29080  |  2.61638
Dataset:          DYE886R   Npts:    15 chi2(cent|diag):     0.43613  |  0.48240
Dataset:          DYE886P   Npts:    89 chi2(cent|diag):     1.43484  |  2.97603

Experiment:           DYE605    Npts:    85 chi2(cent|diag):     1.22235  |  0.78873
Dataset:           DYE605   Npts:    85 chi2(cent|diag):     1.22235  |  0.78873

Experiment:              CDF    Npts:    105    chi2(cent|diag):     1.08185  |  0.29427
Dataset:          CDFZRAP   Npts:    29 chi2(cent|diag):     1.50774  |  0.22786
Dataset:          CDFR2KT   Npts:    76 chi2(cent|diag):     0.92383  |  0.39362

Experiment:               D0    Npts:    45 chi2(cent|diag):     1.16690  |  1.05907
Dataset:           D0ZRAP   Npts:    28 chi2(cent|diag):     0.60486  |  0.60486
Dataset:          D0WEASY   Npts:    8  chi2(cent|diag):     2.73064  |  2.74252
Dataset:          D0WMASY   Npts:    9  chi2(cent|diag):     1.52548  |  0.97578

Experiment:            ATLAS    Npts:    360    chi2(cent|diag):     1.10243  |  1.18744
Dataset:   ATLASWZRAP36PB   Npts:    30 chi2(cent|diag):     0.96138  |  0.75397
Dataset: ATLASZHIGHMASS49FB Npts:    5  chi2(cent|diag):     1.53541  |  1.15946
Dataset: ATLASLOMASSDY11EXT Npts:    6  chi2(cent|diag):     0.89489  |  0.34912
Dataset:     ATLASWZRAP11   Npts:    34 chi2(cent|diag):     2.09671  |  0.19571
Dataset: ATLASR04JETS36PB   Npts:    90 chi2(cent|diag):     0.98864  |  0.94460
Dataset: ATLASR04JETS2P76TEV    Npts:    59 chi2(cent|diag):     1.14861  |  0.89101
Dataset:      ATLAS1JET11   Npts:    31 chi2(cent|diag):     1.12731  |  1.92028
Dataset: ATLASZPT8TEVMDIST  Npts:    44 chi2(cent|diag):     1.05809  |  1.52795
Dataset: ATLASZPT8TEVYDIST  Npts:    48 chi2(cent|diag):     2.62662  |  3.29452
Dataset:    ATLASTTBARTOT   Npts:    3  chi2(cent|diag):     0.87542  |  0.87542
Dataset: ATLASTOPDIFF8TEVTRAPNORM   Npts:    10 chi2(cent|diag):     1.47998  |  1.36148

Experiment:              CMS    Npts:    409    chi2(cent|diag):     1.08602  |  0.64689
Dataset:    CMSWEASY840PB   Npts:    11 chi2(cent|diag):     0.78462  |  0.80156
Dataset:     CMSWMASY47FB   Npts:    11 chi2(cent|diag):     1.73848  |  1.41336
Dataset:        CMSDY2D11   Npts:    110    chi2(cent|diag):     1.27391  |  1.03805
Dataset:       CMSWMU8TEV   Npts:    22 chi2(cent|diag):     1.02757  |  0.46006
Dataset:        CMSJETS11   Npts:    133    chi2(cent|diag):     0.90008  |  0.23938
Dataset:    CMS1JET276TEV   Npts:    81 chi2(cent|diag):     1.05354  |  1.23639
Dataset:       CMSZDIFF12   Npts:    28 chi2(cent|diag):     3.51865  |  0.61532
Dataset:      CMSTTBARTOT   Npts:    3  chi2(cent|diag):     0.19177  |  0.19177
Dataset: CMSTOPDIFF8TEVTTRAPNORM    Npts:    10 chi2(cent|diag):     0.94081  |  0.55425

Experiment:             LHCb    Npts:    85 chi2(cent|diag):     1.56329  |  0.97131
Dataset:       LHCBZ940PB   Npts:    9  chi2(cent|diag):     1.45559  |  0.97615
Dataset:       LHCBZEE2FB   Npts:    17 chi2(cent|diag):     1.15719  |  0.73371
Dataset:     LHCBWZMU7TEV   Npts:    29 chi2(cent|diag):     1.87363  |  1.01114
Dataset:     LHCBWZMU8TEV   Npts:    30 chi2(cent|diag):     1.53178  |  1.06599

- Checking for 4-Sigma deviations from mean
  Replica 10 chi2 is too large: 1.51069
- All replicas tested and verified
- Global average: 1.26088 STD: 0.05946
- Central: 1.15838
Thanks for using LHAPDF 6.2.1. Please make sure to cite the paper:
  Eur.Phys.J. C75 (2015) 3, 132  (http://arxiv.org/abs/1412.7420)

Before change:

Values of chi2 by dataset
-------------------------- 

Experiment:              NMC    Npts:    325    chi2(cent|diag):     1.29935  |  1.00332
Dataset:            NMCPD   Npts:    121    chi2(cent|diag):     0.92692  |  0.90968
Dataset:              NMC   Npts:    204    chi2(cent|diag):     1.52025  |  1.05885

Experiment:             SLAC    Npts:    67 chi2(cent|diag):     0.73120  |  0.68675
Dataset:            SLACP   Npts:    33 chi2(cent|diag):     0.78722  |  0.80192
Dataset:            SLACD   Npts:    34 chi2(cent|diag):     0.68787  |  0.57496

Experiment:            BCDMS    Npts:    581    chi2(cent|diag):     1.20481  |  0.58762
Dataset:           BCDMSP   Npts:    333    chi2(cent|diag):     1.28552  |  0.67808
Dataset:           BCDMSD   Npts:    248    chi2(cent|diag):     1.09679  |  0.46616

Experiment:           CHORUS    Npts:    832    chi2(cent|diag):     1.12325  |  1.09525
Dataset:         CHORUSNU   Npts:    416    chi2(cent|diag):     1.14197  |  1.30722
Dataset:         CHORUSNB   Npts:    416    chi2(cent|diag):     1.05992  |  0.88329

Experiment:           NTVDMN    Npts:    76 chi2(cent|diag):     0.84904  |  0.87437
Dataset:         NTVNUDMN   Npts:    39 chi2(cent|diag):     0.64963  |  0.70954
Dataset:         NTVNBDMN   Npts:    37 chi2(cent|diag):     1.05604  |  1.04811

Experiment:         HERACOMB    Npts:    1145   chi2(cent|diag):     1.15963  |  2.27524
Dataset:     HERACOMBNCEM   Npts:    159    chi2(cent|diag):     1.41558  |  2.43291
Dataset:  HERACOMBNCEP460   Npts:    204    chi2(cent|diag):     1.07406  |  1.97182
Dataset:  HERACOMBNCEP575   Npts:    254    chi2(cent|diag):     0.89981  |  1.49385
Dataset:  HERACOMBNCEP820   Npts:    70 chi2(cent|diag):     1.14477  |  1.55973
Dataset:  HERACOMBNCEP920   Npts:    377    chi2(cent|diag):     1.30151  |  3.26532
Dataset:     HERACOMBCCEM   Npts:    42 chi2(cent|diag):     1.15985  |  1.26672
Dataset:     HERACOMBCCEP   Npts:    39 chi2(cent|diag):     1.13053  |  1.10825

Experiment:      HERAF2CHARM    Npts:    37 chi2(cent|diag):     1.49044  |  0.98288
Dataset:      HERAF2CHARM   Npts:    37 chi2(cent|diag):     1.49044  |  0.98288

Experiment:         F2BOTTOM    Npts:    29 chi2(cent|diag):     1.10977  |  1.17429
Dataset:        H1HERAF2B   Npts:    12 chi2(cent|diag):     0.77584  |  0.43934
Dataset:      ZEUSHERAF2B   Npts:    17 chi2(cent|diag):     1.34549  |  1.69308

Experiment:           DYE886    Npts:    104    chi2(cent|diag):     1.29080  |  2.61638
Dataset:          DYE886R   Npts:    15 chi2(cent|diag):     0.43613  |  0.48240
Dataset:          DYE886P   Npts:    89 chi2(cent|diag):     1.43484  |  2.97603

Experiment:           DYE605    Npts:    85 chi2(cent|diag):     1.22235  |  0.78873
Dataset:           DYE605   Npts:    85 chi2(cent|diag):     1.22235  |  0.78873

Experiment:              CDF    Npts:    105    chi2(cent|diag):     1.08185  |  0.29427
Dataset:          CDFZRAP   Npts:    29 chi2(cent|diag):     1.50774  |  0.22786
Dataset:          CDFR2KT   Npts:    76 chi2(cent|diag):     0.92383  |  0.39362

Experiment:               D0    Npts:    45 chi2(cent|diag):     1.16690  |  1.05907
Dataset:           D0ZRAP   Npts:    28 chi2(cent|diag):     0.60486  |  0.60486
Dataset:          D0WEASY   Npts:    8  chi2(cent|diag):     2.73064  |  2.74252
Dataset:          D0WMASY   Npts:    9  chi2(cent|diag):     1.52548  |  0.97578

Experiment:            ATLAS    Npts:    360    chi2(cent|diag):     1.10243  |  1.18744
Dataset:   ATLASWZRAP36PB   Npts:    30 chi2(cent|diag):     0.96138  |  0.75397
Dataset: ATLASZHIGHMASS49FB Npts:    5  chi2(cent|diag):     1.53541  |  1.15946
Dataset: ATLASLOMASSDY11EXT Npts:    6  chi2(cent|diag):     0.89489  |  0.34912
Dataset:     ATLASWZRAP11   Npts:    34 chi2(cent|diag):     2.09671  |  0.19571
Dataset: ATLASR04JETS36PB   Npts:    90 chi2(cent|diag):     0.98864  |  0.94460
Dataset: ATLASR04JETS2P76TEV    Npts:    59 chi2(cent|diag):     1.14861  |  0.89101
Dataset:      ATLAS1JET11   Npts:    31 chi2(cent|diag):     1.12731  |  1.92028
Dataset: ATLASZPT8TEVMDIST  Npts:    44 chi2(cent|diag):     1.05809  |  1.52795
Dataset: ATLASZPT8TEVYDIST  Npts:    48 chi2(cent|diag):     2.62662  |  3.29452
Dataset:    ATLASTTBARTOT   Npts:    3  chi2(cent|diag):     0.87542  |  0.87542
Dataset: ATLASTOPDIFF8TEVTRAPNORM   Npts:    10 chi2(cent|diag):     1.47998  |  1.36148

Experiment:              CMS    Npts:    409    chi2(cent|diag):     1.08602  |  0.64689
Dataset:    CMSWEASY840PB   Npts:    11 chi2(cent|diag):     0.78462  |  0.80156
Dataset:     CMSWMASY47FB   Npts:    11 chi2(cent|diag):     1.73848  |  1.41336
Dataset:        CMSDY2D11   Npts:    110    chi2(cent|diag):     1.27391  |  1.03805
Dataset:       CMSWMU8TEV   Npts:    22 chi2(cent|diag):     1.02757  |  0.46006
Dataset:        CMSJETS11   Npts:    133    chi2(cent|diag):     0.90008  |  0.23938
Dataset:    CMS1JET276TEV   Npts:    81 chi2(cent|diag):     1.05354  |  1.23639
Dataset:       CMSZDIFF12   Npts:    28 chi2(cent|diag):     3.51865  |  0.61532
Dataset:      CMSTTBARTOT   Npts:    3  chi2(cent|diag):     0.19177  |  0.19177
Dataset: CMSTOPDIFF8TEVTTRAPNORM    Npts:    10 chi2(cent|diag):     0.94081  |  0.55425

Experiment:             LHCb    Npts:    85 chi2(cent|diag):     1.56329  |  0.97131
Dataset:       LHCBZ940PB   Npts:    9  chi2(cent|diag):     1.45559  |  0.97615
Dataset:       LHCBZEE2FB   Npts:    17 chi2(cent|diag):     1.15719  |  0.73371
Dataset:     LHCBWZMU7TEV   Npts:    29 chi2(cent|diag):     1.87363  |  1.01114
Dataset:     LHCBWZMU8TEV   Npts:    30 chi2(cent|diag):     1.53178  |  1.06599

- Checking for 4-Sigma deviations from mean
  Replica 10 chi2 is too large: 1.51069
Zaharid commented 5 years ago

@juanrojochacon @lucarottoli some debugging help would be appreciated here, particularly versions of the code that have and don't have this problem.

scarrazza commented 5 years ago

@tgiani ok, indeed I was looking to the "linearised" master log, the problem seems to appear in 2114409acc442bfa0b98715a54d92c82d0850151

Zaharid commented 5 years ago

@tgiani @wilsonmr The bugged commit might be:

2114409acc442bfa0b98715a54d92c82d0850151

We need to understand why and how to fix it ASAP (as in the next few hours). It is bugged because the chi2 for cmszdiff goes up before and after. This can be seen with chi2check but not validphys, because it uses a different path.

wilsonmr commented 5 years ago

ok so first thing I notice is Nathan adds this line:

if ((isys.name == "THEORYCORR" || isys.name == "THEORYUNCORR") && !use_theory_errors)
              continue; // Continue if systype is theoretical and use_theory_errors == false

so I think we're not using these errors in the construction of the cov mat atm

wilsonmr commented 5 years ago

and the point is that the bugged datasets use these errors right?

voisey commented 5 years ago

In https://github.com/NNPDF/nnpdf/blob/2114409acc442bfa0b98715a54d92c82d0850151/libnnpdf/src/chisquared.cc#L86 it looks like use_theory_errors is hardcoded to be false when perhaps it shouldn't be. @Zaharid tells me that if this is wrong is doesn't have an impact on whether the fit is right or not, but rather it just means that validphys is computing the chi2 per data set incorrectly for data sets with theory errors.

scarrazza commented 5 years ago

@voisey indeed good point.

wilsonmr commented 5 years ago

Sorry, Edinburgh WiFi is down. I can't download any resources to test these things or log onto my university desktop

Zaharid commented 5 years ago

So we need to see how to fix this (presumably by flipping that flag), and also add a test that would have cached this.

wilsonmr commented 5 years ago

Are we sure that this doesn't affect things at the fit level? flipping the flag actually changes the output of chi2check

Zaharid commented 5 years ago

I am not sure, but hopeful. chi2check computes both per-experiment and per-dataset chi². The per-dataset is not used anywhere critical AFAICT.

scarrazza commented 5 years ago

Yes, if you create a single experiment with just the CMSZDIFF12 dataset you can see that it only affects the dataset:

Values of chi2 by dataset
-------------------------- 

Experiment:              CMS    Npts:    28 chi2(cent|diag):     1.32663  |  0.58305
Dataset:       CMSZDIFF12   Npts:    28 chi2(cent|diag):     3.53007  |  0.61776
wilsonmr commented 5 years ago

in fact if you look at

https://vp.nnpdf.science/2PVG3z7YQn-dApSKBUk_4w==/#chi2-by-dataset-comparisons https://vp.nnpdf.science/y_Zrcan3RGyBtn5ZRqwUqQ==/#chi2-by-dataset-comparisons

at the fit in common between reports we see that it is just at the validphys level

EDIT: one report being before the commit and one after

wilsonmr commented 5 years ago

suitable test would probably to have a tagged fit which we calculate chi2 on and it should give the same answer

Zaharid commented 5 years ago

Probably just add something like cmszdiff to the regression tests

On Fri, 7 Dec 2018, 16:50 wilsonmr <notifications@github.com wrote:

suitable test would probably to have a tagged fit which we calculate chi2 on and it should give the same answer

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NNPDF/nnpdf/issues/345#issuecomment-445293863, or mute the thread https://github.com/notifications/unsubscribe-auth/AFabUv80T2XPZsC-8B5l9PFZ8t6fjPgnks5u2pxdgaJpZM4ZIYry .