search

SpeysideHEP / spey

Smooth inference for reinterpretation studies
https://spey.readthedocs.io
MIT License
7 stars 1 forks source link

Inconsistent results with signal uncertainties #37

Closed jackaraz closed 7 months ago

jackaraz commented 7 months ago

System Settings

Name: spey
Version: 0.1.7
Summary: Smooth inference for reinterpretation studies
Home-page: https://github.com/SpeysideHEP/spey
Author: Jack Y. Araz
Author-email: jackaraz@jlab.org
License: MIT
Requires: autograd, numpy, requests, scipy, semantic_version, tqdm
Required-by: spey-pyhf

Platform info:            macOS-14.2.1-arm64-arm-64bit
Python version:           3.9.18
Numpy version:            1.23.5
Scipy version:            1.10.0
Autograd version:         1.5
tqdm version:             4.65.0
semantic_version version: 2.10.0

Installed backend plug-ins:

- pyhf (spey-pyhf-0.1.4)
Name: spey-pyhf
Version: 0.1.4
Summary: pyhf plugin for spey interface
Home-page: https://github.com/SpeysideHEP/spey-pyhf
Author: Jack Y. Araz
Author-email: jackaraz@jlab.org
License: MIT
Requires: pyhf, spey
Required-by: 

- pyhf.simplify (spey-pyhf-0.1.4)
- pyhf.uncorrelated_background (spey-pyhf-0.1.4)
- default_pdf.correlated_background (spey-0.1.7)
- default_pdf.effective_sigma (spey-0.1.7)
- default_pdf.poisson (spey-0.1.7)
- default_pdf.third_moment_expansion (spey-0.1.7)
- default_pdf.uncorrelated_background (spey-0.1.7)

Describe the bug

Exclusion limits with signal uncertainties should be looser, but they are higher.

Thanks to @sabinekraml for pointing this issue out.

Source code

pdf_wrapper = spey.get_backend("default_pdf.uncorrelated_background")
statistical_model = pdf_wrapper(
    signal_yields=[12.0, 15.0],
    background_yields=[50.0,48.0],
    data=[36, 33],
    absolute_uncertainties=[12.0,16.0],
    signal_uncertainty_configuration={"absolute_uncertainties":[3,4]},
    analysis="example",
    xsection=0.123,
)

print(f"1 - CLs: {statistical_model.exclusion_confidence_level()[0]:.5f}")
print(f"POI upper limit: {statistical_model.poi_upper_limit():.5f}")

# 1 - CLs: 0.99563
# POI upper limit: 0.51504

pdf_wrapper = spey.get_backend("default_pdf.uncorrelated_background")
statistical_model = pdf_wrapper(
    signal_yields=[12.0, 15.0],
    background_yields=[50.0,48.0],
    data=[36, 33],
    absolute_uncertainties=[12.0,16.0],
    #signal_uncertainty_configuration={"absolute_uncertainties":[3,4]},
    analysis="example",
    xsection=0.123,
)
print(f"1 - CLs: {statistical_model.exclusion_confidence_level()[0]:.5f}")
print(f"POI upper limit: {statistical_model.poi_upper_limit():.5f}")

# 1 - CLs: 0.97018
# POI upper limit: 0.85633

Tracebacks

No response

Expected behaviour

When signal uncertainties are added, the $\chi^2$ distribution should get wider, but it's getting narrower instead. This might be due to the scaling of the constraint term.

Additional information

No response

Existing GitHub issues