Generate stability plots from meta_args

[millernb]

Thinking forward to correlator fits: we would eventually like to be able to make stability plots where we vary the (1) number of states, (2) the starting time, and (3) the ending time. Fortunately all of these variables can be specified in the meta_args, as well as their minimum and maximum values.

(Ideally we'd like to make stability plots like those in Fig 10 of hep-lat/2011.12166, but I doubt we could generalize this behavior to any lsqfit.nonlinear_fit object.)

Some ideas on how we could implement this feature:

• Automatically make stability plots for all keys in the posterior
• Require an argument to FitGUI (eg, stability_param='E0')

Additional goals:

• Show statistics under value (reduced chi square, Q-value)
• Show Bayes factors when applicable (ie, fit1.y == fit2.y)
• Cache fits rather than regenerate each fit every time the layout is updated (I'll include some example code below).
• Show stability for extrapolations (eg, for a chiral fit, it might be desirable to have a stability plot where we compare the extrapolated values to the physical point while varying the order of xpt corrections)

Cached fit dictionary (haven't tested this, but you get the idea; also, doesn't account for updating priors!)

class FitsDict(dict):
    def __init__(self, fit, fit_setup_function=None, fit_setup_kwargs=None):
        self.data = fit.data
        self.prior = fit.prior
        self.fit_setup_kwargs = fit_setup_kwargs
        self.fit_setup_function = fit_setup_function

    def __str__(self):
        output = ''
        for key in list(self):
            output += '\n---\nFit: '+str(key)+'\n\n'
            output += str(self.__getitem__(key))
        return output

    def __getitem__(self, meta_args): # eg, meta_args = {'n_states' : 3}
        key = tuple((k, meta_args[k]) for k in sorted(meta_args))
        if key not in self:
            super().__setitem__(key, self._make_fit(meta_args))
            return super().__getitem__(key)
        else:
            return super().__getitem__(key)

    def _make_fit(self, meta_args):
        # ... process meta_args, maybe not exactly correct
        setup = {key: meta_args.get(key) or val for key, val in self.fit_setup_kwargs.items()}
        fit = self.fit_setup_function(**setup)
        return fit

[ckoerber]

I see. That is indeed relevant.

I take it you also want to generally have different priors for different meta config values (I mean for shared parameters), right?

The way I would probably go for now, which seems to be on the lines of the dictionary you are implementing right now, would be:

1. Have a y-dictionary where each key corresponds to a specific meta config
2. Have the fit function and prior adopt to the specific meta configs
3. Don't use lsqfitguis meta config setup on runtime but rather "roll-out" all config values into keys such that they are "static" (unless you want to have some meta-meta configs)
4. Generate a new plotting function for the stability plot (which will be updated based on #14)

In other words

from itertools import product

n_exps = range(2, 6)
t_mins = range(0, 5)
PRIOR_KEYS = ["a", "E"]

def make_prior(n, t_min, key):
    return gv.gvar(...)

y = {(n, t_min): Y_DATA[t_min:] for n, t_min in product(n_exps, t_mins)} # assuming Y_DATA is an array
x = {(n, t_min): X_DATA[t_min:] for n, t_min in product(n_exps, t_mins)} # assuming X_DATA is an array
prior = {(n, t_min, key): make_prior(n, t_min, key) for n, t_min, key in product(n_exps, t_mins, PRIOR_KEYS)}

def fcn(x, p):
   out = {}
   for (n, t_min), xx in x.itmes():
       pp = {key: p[(n, t_min, key)] for key in PRIOR_KEYS}
       out[(n, t_min)] = ... # some logic here
   return out

def stability_plot(fit, **kwargs):
   fig = ...
   return fig

fit = nonlinear_fit((x, y), fcn=fcn, prior=prior)
gui = FitGUI(fit)
gui.plots.append({"name": "Stability plot", "fcn": stability_plot, "fcn_kwargs": {...}})

Is this also what you have in mind?

However, this makes caching a bit harder as the extensive computational part is now on the callbacks (is this already meaningful, here?). And, a priori, they do not know which part to re-run and which one does not change the final result. Right now, without having a custom wrapper for the expansive call (namely nonlinear_fit), like the MultiFitterModel provides, I find it quite hard to come up with a general caching model. Happy for suggestionns :)

[millernb]

I'm starting to realize that this might be a bit harder than I originally thought. Even though meta_config contains enough information to uniquely the determine stability plots, actually implementing this feature is going to require an API rewrite.

Using FitsDict (or something akin to it) faces an obvious scope problem. Ultimately we want to access the FitsDict object inside content.py:get_figures since that's where the stability plot will be generate, which means replacing the fit argument with the FitsDict object and a unique key specifying the fit (prior + meta). Tracing backwards from get_figures, that means we'll also need to replace the fit arg in

• content.py:get_content
• content.py:get_layout
• dashboard.py:update_layout_from_meta?
• dashboard.py:update_layout_from_prior?

...and probably other places, too.

A useful starting point might be merging process_meta, process_prior, and fit_setup_function. Something like:

def process_fit(fits_dict, prior_flat=None, prior=None, meta_array=None):
   meta_config = fits_dict.meta_config

   prior =  ... # process prior
   meta_args = .... # process meta_args

   fit_key = ... # generate key from prior, meta_args
   return fits_dict[fit_key]

with most of the real processing happening inside FitsDict._make_fit.

Import edge case to keep in mind: meta_config not specified (eg, run_server(fit))

[millernb]

Alternatively, we could just give up on caching fits -- I suspect we'll create orders of magnitude more fits than we'll retrieve, so the gains might be pretty minor anyway. (But if we're generating many fits, we might want to make liberal use of gv.switch_gvar to prevent slowdowns.)

To make the stability plots, we'll still need access to meta_config and fit_setup_function (and fit_setup_kwargs?) inside content.py:get_figures, however.

[millernb]

Or we could just extend this block of code.

    # from dashboard.py:get_layout

    sidebar = get_sidebar(fit.prior, meta_config=meta_config, meta_values=meta_values)
    sidebar.className = "sticky-top bg-light p-4"

    content = get_content(fit, name=name, plots=plots) if use_default_content else None
    additional_content = get_additional_content(fit) if get_additional_content else None

    layout = html.Div(
        children=html.Div(
            children=[
                html.Div(
                    children=sidebar,
                    className="col-xs-12 col-sm-5 col-md-4 col-xl-3 col-xxl-2",
                    id="sticky-sidebar",
                ),
                html.Div(
                    children=[content, additional_content],
                    className="col-xs-12 col-sm-7 col-md-8 col-xl-9 col-xxl-10",
                ),
            ],
            className="row py-3",
        ),
        className="container-fluid",
    )

For instance, children=[content, additional_content] -> children=[content, additional_content, stability_plots] with another function similar to content.py:get_content except solely responsible for creating the stability plots.

[ckoerber]

In this case, I'd argue stability plots are additional content :)

So get_additional_content would provide the stability plot html. But see also the suggestion in #16

[millernb]

Yup, additional_content is probably the right place for now. I think it's possible to have good defaults such that we could automatically generate stability plots from meta_config, but we should probably write more examples employing meta_config before we attempt to generalize (eg, a multikey example would be nice).

[ckoerber]

Leaving this issue open as the current draft is not yet discussed & finalized.