search

cms-analysis / CombineHarvester

CMSSW package for the creation, editing and analysis of combine datacards and workspaces
cms-analysis.github.io/CombineHarvester/
15 stars 182 forks source link

Investigate post-fit sampling for the cases with high statistics and large lnN uncertainties #297

Open anigamova opened 1 year ago

anigamova commented 1 year ago

Post-fit uncertainty bands are not correctly evaluated from the cov. matrix sampling for bins with high statistics and large lnN uncertainties, reported at CMSTalk.

kcormi commented 1 year ago

This same issue appears for asymmetric gaussian uncertainties with high statistics as well.

As suggested by others, one thing we may want to look into is using the 68% quantile of the sampled toys rather than the variance directly in the computation. I think there may be some other underlying issues in this case that might not be solved, but in general using the asymmetric interval might be a useful change.

ajgilbert commented 1 year ago

See nice summary from @nsmith- here: https://cms-talk.web.cern.ch/t/postfit-uncertainty-bands-very-large/20967/25 Though it doesn't fix the problem, we should align the calculation in CH with the one in FitDiagnostics (subtract from mean instead of best-fit values).

anigamova commented 1 year ago

Thanks @ajgilbert Would this work https://github.com/cms-analysis/CombineHarvester/pull/298 ?