hypotest dosen't converge for ATLAS-SUSY-2019-08

[Ga0l]

Description

Using BkgOnly.json provided with ATLAS-SUSY-2019-08 analysis, it seems pyhf.infer.hypotest doesn't converge and runs indefinitely.

Expected Behavior

hypotest should return a result

Actual Behavior

hypotest runs indefinitely. However, downgrading to pyhf v0.4.4 solves the problem.

Steps to Reproduce

The following code with the last development version of pyhf should reproduce the problem

import pyhf
pyhf.set_backend(b"pytorch")
import json
import jsonpatch

with open('1Lbb-likelihoods-hepdata/BkgOnly.json', 'r') as f:
    bkg = json.load(f)
patch = [{'op': 'add', 'path': '/channels/5/samples/0', 'value': {'data': [0.12533750364832918, 0.17221601055053243, 0.42090085914655334], 'modifiers': [{'data': None, 'type': 'normfactor', 'name': 'mu_SIG'}, {'data': None, 'type': 'lumi', 'name': 'lumi'}], 'name': 'bsm'}},
{'op': 'add', 'path': '/channels/6/samples/0', 'value': {'data': [0.01763733883344263, 0.028793559421499824, 0.1071569435610115], 'modifiers': [{'data': None, 'type': 'normfactor', 'name': 'mu_SIG'}, {'data': None, 'type': 'lumi', 'name': 'lumi'}], 'name': 'bsm'}},
{'op': 'add', 'path': '/channels/7/samples/0', 'value': {'data': [0.004158581794495646, 0.031196622934404727, 0.09260258010973071], 'modifiers': [{'data': None, 'type': 'normfactor', 'name': 'mu_SIG'}, {'data': None, 'type': 'lumi', 'name': 'lumi'}], 'name': 'bsm'}},
{'op': 'remove', 'path': '/channels/4'},
{'op': 'remove', 'path': '/channels/3'},
{'op': 'remove', 'path': '/channels/2'},
{'op': 'remove', 'path': '/channels/1'},
{'op': 'remove', 'path': '/channels/0'}]

llhdSpec = jsonpatch.apply_patch(bkg, patch)
msettings = {'normsys': {'interpcode': 'code4'}, 'histosys': {'interpcode': 'code4p'}}
workspace = pyhf.Workspace(llhdSpec)
model = workspace.model(modifier_settings=msettings)
result = pyhf.infer.hypotest( 10., workspace.data(model), model, qtilde=True, return_expected=False)
print(result)

downgrading pyhf with python -m pip install --user pyhf==0.4.4 solves the issue

Checklist

• [x] Run python -m pip install --ignore-installed -U --user "git+https://github.com/scikit-hep/pyhf.git#egg=pyhf[torch]"
• [x] Searched through existing Issues to confirm this is not a duplicate issue
• [x] Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue

[matthewfeickert]

We'll have to follow up more on this given that if you add after the model definition

import numpy as np

for test_mu in np.arange(7).tolist():
    print(f"test_mu: {test_mu}")
    result = pyhf.infer.hypotest(
        test_mu, workspace.data(model), model, qtilde=True, return_expected=False
    )
    print(result)

and then run the output is

$ python example.py
test_mu: 0
tensor(nan)
test_mu: 1
tensor(0.8648)
test_mu: 2
tensor(0.7694)
test_mu: 3
tensor(0.6993)
test_mu: 4
tensor(0.6450)
test_mu: 5
tensor(0.5938)
test_mu: 6

and then it gets caught up on test_mu=6. I suspect that this has more to do with the scaling of the model then anything else, but we can follow up more.