cms-analysis / HiggsAnalysis-CombinedLimit

CMS Higgs Combination toolkit.
https://cms-analysis.github.io/HiggsAnalysis-CombinedLimit/latest
Apache License 2.0
75 stars 380 forks source link

Poor error reporting from AsymptoticLimits with bad fits #925

Open andrzejnovak opened 5 months ago

andrzejnovak commented 5 months ago

I have a workspace for which the expected limits fits are currently failing kind of silently by just not printing the expected results

╰─ff combine -M AsymptoticLimits -d model_combined.root
 <<< Combine >>>
>>> random number generator seed is 123456
>>> method used is AsymptoticLimits
SimNLL created with 15 channels, 0 generic constraints, 181 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,
SimNLL created with 15 channels, 0 generic constraints, 181 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,
SimNLL created with 15 channels, 0 generic constraints, 181 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,

 -- AsymptoticLimits ( CLs ) --
Observed Limit: r < 0.6503

Done in 16.02 min (cpu), 16.02 min (real)

In the meantime, while looking at available args to debug, and came across --picky Abort on fit failures which does not actually abort, but instead prints the garbage expected limit along with extra errors

╰─ff combine -M AsymptoticLimits -d model_combined.root --picky
 <<< Combine >>>
>>> random number generator seed is 123456
>>> method used is AsymptoticLimits
SimNLL created with 15 channels, 0 generic constraints, 181 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,
SimNLL created with 15 channels, 0 generic constraints, 181 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,
SimNLL created with 15 channels, 0 generic constraints, 181 fast gaussian constraints, 0 fast poisson constraints, 0 fast group constraints,
Minimization failed in an unrecoverable way
Minimization failed in an unrecoverable way

 -- AsymptoticLimits ( CLs ) --
Observed Limit: r < 1.2718
Expected  2.5%: r < 0.0078
Expected 16.0%: r < 0.0087
Expected 50.0%: r < 0.0625
Expected 84.0%: r < 0.0665
Expected 97.5%: r < 0.0670

Done in 18.35 min (cpu), 18.37 min (real)

This would ideally report more meaningful errors.