search

SpeysideHEP / spey

Smooth inference for reinterpretation studies
https://spey.readthedocs.io
MIT License
8 stars 1 forks source link

optimiser argument ntrial > 1 crashes #24

Closed WolfgangWaltenberger closed 11 months ago

WolfgangWaltenberger commented 11 months ago

System Settings

In [2]: spey.about()
Name: spey
Version: 0.1.3
Summary: Smooth inference for reinterpretation studies
Home-page: https://github.com/SpeysideHEP/spey
Author: Jack Y. Araz
Author-email: jack.araz@durham.ac.uk
License: MIT
Location: /home/walten/.venvs/311/lib/python3.11/site-packages
Requires: autograd, numpy, scipy, semantic-version, tqdm
Required-by: spey-pyhf

Platform info:            Linux-6.6.1-060601-generic-x86_64-with-glibc2.38
Python version:           3.11.6
Numpy version:            1.26.2
Scipy version:            1.10.0
Autograd version:         1.5
tqdm version:             4.66.1
semantic_version version: 2.10.0

Installed backend plug-ins:

- default_pdf.correlated_background (spey-0.1.3)
- default_pdf.effective_sigma (spey-0.1.3)
- default_pdf.poisson (spey-0.1.3)
- default_pdf.third_moment_expansion (spey-0.1.3)
- default_pdf.uncorrelated_background (spey-0.1.3)
- pyhf (spey-pyhf-0.1.1)
Name: spey-pyhf
Version: 0.1.1
Summary: pyhf plugin for spey interface
Home-page: https://github.com/SpeysideHEP/spey-pyhf
Author: Jack Y. Araz
Author-email: jack.araz@durham.ac.uk
License: MIT
Location: /home/walten/.venvs/311/lib/python3.11/site-packages
Requires: pyhf, spey
Required-by: 

- pyhf.simplify (spey-pyhf-0.1.1)
- pyhf.uncorrelated_background (spey-pyhf-0.1.1)

Describe the bug

if i want to call the optimiser with ntrial > 1, spey crashes

Source code

#!/usr/bin/env python3

obsN=[138.0, 91.0, 14.0, 3.0, 54.0, 38.0, 4.0, 0.0, 8.0, 2.0, 4.0, 0.0, 1.0, 3.0, 1.0, 1.0, 42.0, 6.0, 1.0, 4.0, 0.0, 0.0]
bg=[160.57, 90.439, 11.512, 2.7508, 53.529, 28.319, 2.5856, 2.6029, 5.0628, 2.1711, 0.063528, 0.88532, 2.6812, 1.2599, 0.41911, 0.67201, 33.567, 7.3285, 1.6546, 4.0289, 0.88189, 0.19966]
cov=[[174.5, 6.601, 0.2403, 0.163, 19.64, 6.506, 0.5148, 0.519, 1.754, 0.6346, 0.007154, 0.422, 1.397, 0.6174, 0.171, 0.286, -0.5543, -0.1305, 0.02306, -0.05027, -0.006686, 0.0109], [6.601, 89.09, 0.5931, -0.04077, 7.188, 7.695, 0.3474, 0.3852, 1.01, 0.4502, 0.009081, 0.2456, 0.8593, 0.4282, 0.1174, 0.235, -0.2041, -0.04318, 0.03952, -0.1545, -0.01002, 0.01648], [0.2403, 0.5931, 9.672, -0.01319, 0.5717, 0.4039, 0.4499, 0.03413, 0.08148, 0.03547, 0.001684, 0.07249, 0.07084, 0.04303, 0.01724, 0.02002, 0.03696, 0.0385, 0.02107, -0.01743, 0.001832, 0.003709], [0.163, -0.04077, -0.01319, 3.709, 0.2659, 0.1292, -0.001839, 0.5376, 0.03348, 0.00997, 0.001397, 0.05909, 0.01182, 0.005355, 0.006677, 0.03921, 0.01953, -0.01931, 0.02678, 0.01732, -0.002887, 0.004678], [19.64, 7.188, 0.5717, 0.2659, 69.77, 6.934, 0.5394, 0.8391, 1.98, 0.7593, 0.01766, 0.3011, 1.661, 0.7474, 0.216, 0.224, -0.2935, -0.01572, 0.04757, -0.001996, 0.01541, 0.01112], [6.506, 7.695, 0.4039, 0.1292, 6.934, 27.89, 0.3204, 0.3368, 1.036, 0.4105, 0.008852, 0.1561, 0.8648, 0.3931, 0.1036, 0.1509, -0.2699, -0.02976, 0.006113, -0.06488, 0.01266, 0.01277], [0.5148, 0.3474, 0.4499, -0.001839, 0.5394, 0.3204, 1.699, 0.01839, 0.08511, 0.02799, 0.0003195, 0.001698, 0.06698, 0.02765, 0.01019, 0.004009, -0.06498, -0.01117, 0.01824, 0.01453, 0.002514, 0.002796], [0.519, 0.3852, 0.03413, 0.5376, 0.8391, 0.3368, 0.01839, 5.76, 0.09795, 0.05031, 0.002012, 0.04714, 0.078, 0.03875, 0.01596, 0.02729, 0.0001083, -0.01461, 0.01978, 0.02572, 0.004099, 0.0067], [1.754, 1.01, 0.08148, 0.03348, 1.98, 1.036, 0.08511, 0.09795, 2.108, 0.1177, 0.002927, 0.03125, 0.5298, 0.1115, 0.02586, 0.03543, 0.03278, -0.006467, 0.005036, 0.002551, -0.002566, 0.0006324], [0.6346, 0.4502, 0.03547, 0.00997, 0.7593, 0.4105, 0.02799, 0.05031, 0.1177, 0.5019, 0.001326, 0.0148, 0.09351, 0.1455, 0.01469, 0.01873, 0.02184, 0.002662, 0.00445, -0.004747, 0.0002134, 0.0007696], [0.007154, 0.009081, 0.001684, 0.001397, 0.01766, 0.008852, 0.0003195, 0.002012, 0.002927, 0.001326, 0.0121, 0.0001844, 0.001437, 0.001037, 0.01401, 0.0009814, -0.0007443, -8.312e-05, 0.0009001, 0.0009795, 0.0001779, 0.0002314], [0.422, 0.2456, 0.07249, 0.05909, 0.3011, 0.1561, 0.001698, 0.04714, 0.03125, 0.0148, 0.0001844, 4.761, 0.03474, 0.01384, 0.007457, 1.983, 0.07728, 0.0002071, 0.009179, 0.07125, 0.007518, 0.004858], [1.397, 0.8593, 0.07084, 0.01182, 1.661, 0.8648, 0.06698, 0.078, 0.5298, 0.09351, 0.001437, 0.03474, 0.8703, 0.0914, 0.02294, 0.02833, 0.009643, -0.001688, 0.003651, -0.005076, -0.0001599, 0.001133], [0.6174, 0.4282, 0.04303, 0.005355, 0.7474, 0.3931, 0.02765, 0.03875, 0.1115, 0.1455, 0.001037, 0.01384, 0.0914, 0.2891, 0.01253, 0.01471, 0.00162, 0.002663, 0.00349, -0.002747, -0.001798, -0.0003085], [0.171, 0.1174, 0.01724, 0.006677, 0.216, 0.1036, 0.01019, 0.01596, 0.02586, 0.01469, 0.01401, 0.007457, 0.02294, 0.01253, 0.3721, 0.005811, 0.006178, 0.004303, 0.001909, 0.002492, 0.0006547, 0.0008085], [0.286, 0.235, 0.02002, 0.03921, 0.224, 0.1509, 0.004009, 0.02729, 0.03543, 0.01873, 0.0009814, 1.983, 0.02833, 0.01471, 0.005811, 1.776, 0.06913, 0.008452, 0.01022, 0.03104, 0.007245, 0.005361], [-0.5543, -0.2041, 0.03696, 0.01953, -0.2935, -0.2699, -0.06498, 0.0001083, 0.03278, 0.02184, -0.0007443, 0.07728, 0.009643, 0.00162, 0.006178, 0.06913, 33.32, 5.406, 1.608, 1.192, 0.0703, 0.07672], [-0.1305, -0.04318, 0.0385, -0.01931, -0.01572, -0.02976, -0.01117, -0.01461, -0.006467, 0.002662, -8.312e-05, 0.0002071, -0.001688, 0.002663, 0.004303, 0.008452, 5.406, 3.271, 0.5664, 0.07033, 0.262, 0.03766], [0.02306, 0.03952, 0.02107, 0.02678, 0.04757, 0.006113, 0.01824, 0.01978, 0.005036, 0.00445, 0.0009001, 0.009179, 0.003651, 0.00349, 0.001909, 0.01022, 1.608, 0.5664, 0.8431, 0.07743, 0.03581, 0.1028], [-0.05027, -0.1545, -0.01743, 0.01732, -0.001996, -0.06488, 0.01453, 0.02572, 0.002551, -0.004747, 0.0009795, 0.07125, -0.005076, -0.002747, 0.002492, 0.03104, 1.192, 0.07033, 0.07743, 1.903, 0.203, 0.01753], [-0.006686, -0.01002, 0.001832, -0.002887, 0.01541, 0.01266, 0.002514, 0.004099, -0.002566, 0.0002134, 0.0001779, 0.007518, -0.0001599, -0.001798, 0.0006547, 0.007245, 0.0703, 0.262, 0.03581, 0.203, 0.13, 0.04504], [0.0109, 0.01648, 0.003709, 0.004678, 0.01112, 0.01277, 0.002796, 0.0067, 0.0006324, 0.0007696, 0.0002314, 0.004858, 0.001133, -0.0003085, 0.0008085, 0.005361, 0.07672, 0.03766, 0.1028, 0.01753, 0.04504, 0.04159]]
nsig=[0.0, 0.0002892207, 0.0004011771, 0.005287607475, 0.000170267025, 0.000746376, 0.0007673678249999999, 0.0069832804500000005, 0.0007930245, 0.00267295905, 0.001835618475, 0.031107552225, 0.0008769918, 0.005873046149999999, 0.007456762725000001, 0.085919539725, 0.59161307046, 0.7652938326899998, 5.264143279499999, 0.5127043338, 0.63509786919, 4.421279522100001]
analysis='CMS-SUS-20-004'
lumi=137.0

import spey

stat_wrapper = spey.get_backend("default_pdf.correlated_background")           
speyModel = stat_wrapper( data = obsN, background_yields = bg,
    covariance_matrix = cov, signal_yields = nsig,
    xsection = [ x / lumi for x in nsig ], analysis = analysis )

print ( f"spey oUL(mu)={speyModel.poi_upper_limit( optimiser_arguments = { 'ntrial': 3 } ):.4f}" ) 
# print ( f"spey eUL(mu)={speyModel.poi_upper_limit( expected = spey.ExpectationType.aposteriori ):.4f}" )

Tracebacks

sonic ~/Downloads> python3 forjack.py 
/home/walten/.venvs/311/lib/python3.11/site-packages/spey/optimizer/scipy_tools.py:57: OptimizeWarning: Unknown solver options: ntrial
  opt = scipy.optimize.minimize(
minimization [0] 67.59623551652393 Optimization terminated successfully
/home/walten/.venvs/311/lib/python3.11/site-packages/spey/optimizer/scipy_tools.py:57: OptimizeWarning: Unknown solver options: ntrial
  opt = scipy.optimize.minimize(
minimization [1] 67.59630122863426 Optimization terminated successfully
/home/walten/.venvs/311/lib/python3.11/site-packages/spey/optimizer/scipy_tools.py:57: OptimizeWarning: Unknown solver options: ntrial
  opt = scipy.optimize.minimize(
minimization [2] 58.22544816184985 Optimization terminated successfully
_prepare for llhd 3.1290050479353755e-14 0.0
minimization [3] 67.59630122863426 Optimization terminated successfully
minimization [4] 58.22544816184985 Optimization terminated successfully
minimization [5] 67.59630122863426 Optimization terminated successfully
/home/walten/.venvs/311/lib/python3.11/site-packages/spey/optimizer/scipy_tools.py:73: RuntimeWarning: Inequality constraints incompatible
spey::Expanding the bounds.
  warnings.warn(
Traceback (most recent call last):
  File "/home/walten/Downloads/forjack.py", line 17, in <module>
    print ( f"spey oUL(mu)={speyModel.poi_upper_limit( optimiser_arguments = { 'ntrial': 3 } ):.4f}" ) 
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/base/hypotest_base.py", line 898, in poi_upper_limit
    sigma_mu = self.sigma_mu(muhat, expected=expected)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/base/hypotest_base.py", line 522, in sigma_mu
    qmuA = teststat_func(poi_test, muhatA, -min_nllA, logpdf_asimov)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/hypothesis_testing/test_statistics.py", line 41, in qmu_tilde
    -2.0 * (logpdf(mu) - (max_logpdf if muhat >= 0.0 else logpdf(0.0))), 0.0, None
            ^^^^^^^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/base/hypotest_base.py", line 515, in logpdf_asimov
    return -self.asimov_likelihood(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/interface/statistical_model.py", line 446, in asimov_likelihood
    return self.likelihood(
           ^^^^^^^^^^^^^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/interface/statistical_model.py", line 305, in likelihood
    logpdf, _ = fit(
                ^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/optimizer/core.py", line 52, in fit
    fun, x = minimize(
             ^^^^^^^^^
  File "/home/walten/.venvs/311/lib/python3.11/site-packages/spey/optimizer/scipy_tools.py", line 82, in minimize
    bounds[bdx] = (min(bound[0], bound[0] * 10.0), bound[1] * 10.0)
                                 ~~~~~~~~~^~~~~~
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

Expected behaviour

No response

Additional information

No response

Existing GitHub issues

jackaraz commented 11 months ago

Hi @WolfgangWaltenberger it's "ntrials" not "ntrial". Once this is changed your code runs perfectly without an issue for me.