pyhf / pyhf-tutorial

Updating tutorial on pyhf
https://pyhf.github.io/pyhf-tutorial/
Apache License 2.0
9 stars 5 forks source link

Pseudo-experiments notebook is giving `nan`s #63

Closed matthewfeickert closed 1 year ago

matthewfeickert commented 1 year ago

Following PR #61, in the Playing with Toys notebook, in the Hypothesis Testing (low stats) section using the toybased calculator is yielding nans in the calculation:

CLs_obs, CLs_exp = pyhf.infer.hypotest(
    1.0,  # null hypothesis
    [5.0, 7.0] + model.config.auxdata,
    model,
    test_stat="qtilde",
    return_expected_set=True,
    calctype="toybased",
    ntoys=1000,
    track_progress=False,
)
print(f"      Observed CLs: {CLs_obs:.4f}")
for expected_value, n_sigma in zip(CLs_exp, np.arange(-2, 3)):
    print(f"Expected CLs({n_sigma:2d} σ): {expected_value:.4f}")
/usr/local/venv/lib/python3.10/site-packages/scipy/optimize/_optimize.py:353: RuntimeWarning: Values in x were outside bounds during a minimize step, clipping to bounds
  warnings.warn("Values in x were outside bounds during a "
      Observed CLs: nan
Expected CLs(-2 σ): 0.0000
Expected CLs(-1 σ): 0.0452
Expected CLs( 0 σ): 0.1756
Expected CLs( 1 σ): 0.4375
Expected CLs( 2 σ): 1.0000
/usr/local/venv/lib/python3.10/site-packages/pyhf/infer/calculators.py:864: RuntimeWarning: invalid value encountered in divide
  CLs = tensorlib.astensor(CLsb / CLb)
matthewfeickert commented 1 year ago

Okay running this locally with track_progress=True I can see this is because the signal-like pseudo-experiments fail after less than 200 trials

Signal-like:  16%|████████████████████▌                                                                                                            | 159/1000 [00:02<00:12, 66.83toy/s]
/home/feickert/.pyenv/versions/pyhf-tutorial-dev/lib/python3.10/site-packages/scipy/optimize/_optimize.py:353: RuntimeWarning: Values in x were outside bounds during a minimize step, clipping to bounds
  warnings.warn("Values in x were outside bounds during a "