icecube / pisa

Monte Carlo-based data analysis
http://icecube.github.io/pisa/
Apache License 2.0
19 stars 47 forks source link

Add staged fit #685

Closed atrettin closed 2 years ago

atrettin commented 2 years ago

Add a module to perform a staged fit, where a later fit starts from the best fit point of an earlier fit.

The use case for this is primarily to polish the result of a global search. For example, I can run NLOPT's CRS2 global search algorithm for up to 30h and then polish the best fit point using L-BFGS-B with the following kwargs to fit_recursively():

method="staged",
method_kwargs=None,
# List of subsidiary fits must at least have length 2.
# Fits are run in order of the list.
local_fit_kwargs=[
    fit_nlopt_crs2,
    local_fit_scipy,
]

where

fit_nlopt_crs2 = collections.OrderedDict(
    method="nlopt",
    method_kwargs={
        "algorithm": "NLOPT_GN_CRS2_LM",
        "ftol_rel": 1e-7,
        "ftol_abs": 1e-5,
        "maxtime": 30 * 3600,
    },
    local_fit_kwargs=None,
)

and

scipy_settings_l_bfgs_b = {
  "method": {
    "value": "L-BFGS-B",
    "desc": "The string to pass to scipy.optimize.minimize so it knows what to use"
  },
  "options":{
    "value": {
      "disp"   : 0,
      "ftol"   : 1.0e-7,
      "eps"    : 1.0e-7,
      "gtol"   : 1.0e-8,
      "maxiter": 10000
    },
    "desc": {
      "disp"   : "Set to True to print convergence messages",
      "ftol"   : "Precision goal for the value of f in the stopping criterion",
      "eps"    : "Step size used for numerical approximation of the jacobian.",
      "maxiter": "Maximum number of iteration"
    }
  },
}
local_fit_scipy = {
    "method": "scipy",
    "method_kwargs": scipy_settings_l_bfgs_b,
    "local_fit_kwargs": None
}

One issue that I hope won't cause problems is the fact that I set the nominal parameters of the hypo_maker to the best fit values of the previous fit. This is done to make sure that a subsidiary fit doesn't destroy all progress by calling reset_free(), which could easily happen by mistake since it is the default for some modules. We should test whether or not this creates problems with different param selections similar to what @ts4051 has found elsewhere.

atrettin commented 2 years ago

Rebased onto master. I discovered a bug after including the changes from the latest merge: When one of the fits in the staged fit is an octant-fit (fit_octants), then it crashes at L1397 in analysis.py because the range of the angle parameter was not reset.

atrettin commented 2 years ago
atrettin commented 2 years ago

Finally, unit tests are working! This was a huge technical debt on my side, but now we can have at least a little bit of confidence in this thing. :)

atrettin commented 2 years ago

Tiny addition: Exposing the tolerance setting for octant fits. In some cases where the likelihood at the inflection point is very flat and the truth is far away, I might want to offset a little more into the octant (or quadrant in this case, the pathological angle is the sterile CP phase with inflection point at 90 degrees).