hypotest documentation not clear on return types depending on test statistic chosen

[kratsg]

Summary

If we have return_tail_probs=True, then for test_stat in ['qmu', 'qtilde'], we return [CLs, CLsb, CLb]; while for test_stat=q0 we return [CLsb, CLb] only.

Documentation Page Link

https://scikit-hep.org/pyhf/_generated/pyhf.infer.hypotest.html#pyhf.infer.hypotest

Code of Conduct

• [X] I agree to follow the Code of Conduct

[kratsg]

To summarize briefly from offline with @phinate and @alexander-held (some bullet points):

• https://github.com/scikit-hep/cabinetry/blob/59b49fbb5e3caffb76b12df5277d71912e909712/src/cabinetry/fit/__init__.py#L1106-L1115 gives p0 for reference
• https://github.com/gradhep/relaxed/blob/2bff24096f06068eba9b6ec75fdb0ab69a1243bc/src/relaxed/infer.py#L137
• conventionally, we don't typically report CLs for discovery, only $p0$ and $p\mu$
• likely $\mathrm{CL}_{sb} \equiv p_0$ in pyhf's code base but this needs to be clarified a lot
• CL naming is just bad i think everyone agrees on this

[kratsg]

One more thing, CLx is making the implicit assumption (for x=s, sb, b) that the test stat involve is either qmu or qmu-tilde.

[matthewfeickert]

One more thing, CLx is making the implicit assumption (for x=s, sb, b) that the test stat involve is either qmu or qmu-tilde.

We do state this in the pyhf.infer.hypotest docs

https://github.com/scikit-hep/pyhf/blob/852a224d21da19ed25b7a8548315342d3c7c0c53/src/pyhf/infer/__init__.py#L108-L112

with a special note as well

https://github.com/scikit-hep/pyhf/blob/852a224d21da19ed25b7a8548315342d3c7c0c53/src/pyhf/infer/__init__.py#L115-L122

Although the place that I've found it to be the most clear (and I should update this as well in the docs) is in Appendix A.1.2 and Table 11 in the Procedure for the LHC Higgs boson search combination in Summer 2011 — which actually describes what ATLAS and CMS did for the Higgs discovery.

(This reference document has been quite useful before, as we used it as a reference check in PR https://github.com/scikit-hep/pyhf/pull/1610).

[matthewfeickert]

CL naming is just bad i think everyone agrees on this

@cranmer would agree with you too, I think, given Issue https://github.com/scikit-hep/pyhf/issues/965. :)