search

NNPDF / nnpdf

An open-source machine learning framework for global analyses of parton distributions.
https://docs.nnpdf.science/
GNU General Public License v3.0
30 stars 6 forks source link

Investigate symmetrisation of errors #899

Closed nhartland closed 8 months ago

nhartland commented 6 years ago

In principle we should be including a shift of the central value when symmetrising errors. This is for the most part being done in filters, but some (as far as I can tell just jets) are ignoring the shift.

This may or may not have been done deliberately. We should check the impact that this has.

nhartland commented 6 years ago

OK @lucarottoli @enocera , here is the list of filters which use the symmetriseErrors function.

All filters that call symmetriseErrors

ATLASR04JETS36PB ATLASR06JETS36PB ATLASR04JETS2P76TEV ATLAS1JET11 ATLASPHT15 ATLASPHT12 CDFR2KT CMSJETS11 CMS1JET276TEV D0ZRAP D0R2CON CMSTTBARTOT Z06NC Z06CC ZEUSHERA2CCP ZEUSHERA2NCP ZEUSHERAF2B ATLASTOPDIFF8TEVTPT ATLASTOPDIFF8TEVTRAP ATLASTOPDIFF8TEVTTRAP ATLASTOPDIFF8TEVTTPT ATLASTOPDIFF8TEVTTM ATLASTOPDIFF8TEVTPTNORM ATLASTOPDIFF8TEVTRAPNORM ATLASTOPDIFF8TEVTTRAPNORM ATLASTOPDIFF8TEVTTPTNORM ATLASTOPDIFF8TEVTTMNORM CMSTOPDIFF8TEVTPT CMSTOPDIFF8TEVTRAP CMSTOPDIFF8TEVTTRAP CMSTOPDIFF8TEVTTPT CMSTOPDIFF8TEVTTM

Filters with same-sign up and down systematics (deliberate and correct)

ATLASR04JETS36PB ATLASR06JETS36PB ATLASR04JETS2P76TEV

Filters with same-sign up and down systematics (bugged due to CMSTOPDIFF8TEVTOT)

CMSTOPDIFF8TEVTPT CMSTOPDIFF8TEVTRAP CMSTOPDIFF8TEVTTRAP CMSTOPDIFF8TEVTTPT CMSTOPDIFF8TEVTTM

nhartland commented 6 years ago

The ATLAS jets are then genuinely bugged, along with CMSTOPDIFF8TEV (unnormalised). The rest aren't really bugged, but use an unnecessarily approximate form of D'Agostini's suggestion, which is a poor approximation when uncertainties are very asymmetric.

We should

  1. Fix the CMSTOPDIFF8TEVTOT downwards systematic (add a minus sign) - Done, to be checked by @enocera

  2. Figure out if the ATLAS jets have genuine same-sign systematics or it's just a mistake. Fix it either way - It's genuine

  3. Use the full expression for the symmetrised uncertainties for everything else. done

  4. (possibly) re-introduce shifts in the case of jet data, as they would then have larger uncertainties due to (3)

nhartland commented 6 years ago

Oh yes, and

  1. Check that the treatment of symmetrised uncertainties and shifts in terms of multiplicative uncertainties is correct.
enocera commented 6 years ago

Thanks for the list. I can take care of 1. and 5. (for all the experiments shown above), if you agree.

nhartland commented 6 years ago

Sounds good, I'll look after 2 and 3.

lucarottoli commented 6 years ago

Hi Emanuele, Nathan If Emanuele takes care of 1, I can take care of 5. I guess that for 4 we need to discuss what do to in future fits.

nhartland commented 6 years ago

Yeah, 4 sort of depends on whether or not 3 fixes the original problem.

nhartland commented 6 years ago

Anyone mind if I just remove D0R2CON, Z06NC, Z06CC, ZEUSHERA2CCP, ZEUSHERA2NCP?

In terms of datasets we no longer use I was working under the general strategy of leaving them be if they're cooperating (why not after all) and removing them at the first sight of trouble.

This would probably count as trouble, so I'll probably remove them.

juanrojochacon commented 6 years ago

Please delete all of these. I would also delete all old filters, unless we have a good reason to think that we will want to add back this specific experiment into the fit again.

scarlehoff commented 8 months ago

I'm closing this @enocera but this is a point to keep in mind as new data is implemented, if only not to repeat the errors of the past.