search

icecube / pisa

Monte Carlo-based data analysis
http://icecube.github.io/pisa/
Apache License 2.0
19 stars 47 forks source link

Possible bug in `mceff_llh` #719

Closed BenSmithers closed 1 year ago

BenSmithers commented 1 year ago

I carried out two sensitivity scans for a central/asimov expectation (no fluctuations in bins from their nominal values), only changing the metric, and get wildly different results. A standard Poisson LLH gives something similar to what we expect, and have seen in previous implementations (GolemFit). However, using the effective likelihood (mceff_llh) produces contours that not only mimic signal, but also exclude the nominal/central expectation at >90% CL.

I've been poking around in the code, but can't seem to find the cause of this.

BenSmithers commented 1 year ago

This is actually functioning as designed. Poor MC statistics in a test can lead to substantial bias with this likelihood function