Changing the Tuple of Indices for GridLocator in PlotPrefs and True Trajectory is still different in some cases

[yb6599]

The experiments fail because of the PlotPrefs GridLocator Tuple of Indices (2, 3, 4). When I change it to (1, 2, 3), it runs fine.

    "absrel-newloc": _PlotPrefs(
        True,
        False,
        GridLocator(
            ["coeff_mse", "coeff_f1"],
            (..., (2, 3, 4)),
            (
                {"diff_params.kind": "kalman", "diff_params.alpha": None},
                {
                    "diff_params.kind": "kalman",
                    "diff_params.alpha": lambda a: isinstance(a, int),
                },
                {"diff_params.kind": "trend_filtered"},
                {"diff_params.diffcls": "SmoothedFiniteDifference"},
            ),
        ),
    )

However, the True Trajectory figures are still different, for cubic_ho:

[Jacob-Stevens-Haas]

My guess is that that's because you're not using at least 5 values of sim_params.noise_rel. Which values for noise rel did you use?

This is a really interesting example, since it looks like the initial values are the same, the second just takes more steps in the integration. Can you try to add a logging message (logger.info(...)) when simulating to see what point is simulated, what x0 was, and how long the simulation went?

[yb6599]

sim_params.noise_rel are [0.05, 0.1, 0.15, 0.2, 0.25, 0.3]

I double checked again and you are right, they all start from the same initial point and the length of the trajectory depends on how long the simulation went for. I'll still add a logging message and run the experiment again.

[Jacob-Stevens-Haas]

Interesting, then I'm not sure how you're getting a ValueError for (2, 3, 4). Can you show me a more full error?

[yb6599]

The error goes like this

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[3], line 1
----> 1 results = ex.run(**resolved_args)

File ~/gen-experiments/src/gen_experiments/gridsearch/__init__.py:258, in run(seed, group, grid_params, grid_vals, grid_decisions, other_params, skinny_specs, series_params, metrics, plot_prefs)
    236 results: GridsearchResultDetails = {
    237     "system": group,
    238     "plot_data": [],
   (...)
    255     ),
    256 }
    257 if plot_prefs:
--> 258     plot_data = find_gridpoints(plot_prefs.plot_match, intermediate_data, results)
    259     results["plot_data"] = plot_data
    260     for gridpoint in plot_data:

File ~/gen-experiments/src/gen_experiments/gridsearch/__init__.py:601, in find_gridpoints(find, where, context)
    599     for index_of_ax, indexes_in_ax in keep_axes:
    600         amax_arr = ser[index_of_ax][1]
--> 601         amax_want = amax_arr[np.ix_(inds_of_metrics, indexes_in_ax)].flatten()
    602         partial_match |= {void_to_tuple(el) for el in amax_want}
    603 logger.info(
    604     f"Found {len(partial_match)} gridpoints that match metric-plot_axis criteria"
    605 )

IndexError: index 4 is out of bounds for axis 1 with size 4

Question: What is the significance of the tuple of indices (2, 3, 4) and can they not just be replaced by (1, 2, 3)?

[Jacob-Stevens-Haas]

2, 3, 4 are the indices of the plot axis that we're looking for. So if sim_params.noise_rel are [0.05, 0.1, 0.15, 0.2, 0.25, 0.3], then (2, 3, 4) refers to 0.15, .2, and 0.25. There's nothing inherently important about them, but the fact that it's getting a bug is concerning.

[Jacob-Stevens-Haas]

Ok I've recreated this error. This is because sim_params.t_end only has four grid values, indexed at 0-3. The way forwards is to either (a) add a value to grid_vals[0] so that index 4 exists or change the keep_axis argument to plot_prefs.plot_match

        keep_axis: The grid-varied parameter in which to find results and which
            index of values for that parameter.  To search a particular value of
            that parameter, use the param_match kwarg.It can be specified in
            several ways:
            (a) a tuple of two ellipses, representing all axes, all indices
            (b) a tuple of an ellipsis, representing all axes, and a tuple of
                ints for specific indices
            (c) a tuple of a tuple of strings for specific axes, and an ellipsis
                for all indices
            (d) a collection of tuples of a string (specific axis) and tuple of
                ints (specific indices)

The fourth form allows different indices for different plot axes (t_end and rel_noise).

FWIW, here's the launch.json config that I'm using:

        {
            "name": "Grid None",
            "type": "debugpy",
            "request": "launch",
            "module": "mitosis",
            "justMyCode": false,
            "args": [
                "gen_experiments.gridsearch",
                "-g",
                "none",
                "-F",
                "trials/debug",
                "-e",
                "seed=15",
                "--debug",
                "--param",
                "metrics=all",
                "--param",
                "other_params=4nonzero-cubic",
                "--param",
                "grid_params=rel_noise",
                "--param",
                "grid_vals=rel_noise",
                "--param",
                "grid_decisions=plot2",
                "--param",
                "series_params=kalman-auto3",
                "--param",
                "+plot_prefs=absrel-newloc",
                "--param",
                "+skinny_specs=duration-noise"
            ]
        },

This is an empty experiment primarily for testing the gridseach functionality.