Add functions for post-mortem plotting across experiments

[Jacob-Stevens-Haas]

This pull requests adds several types to utils.py that describe aspects of data saved during a gridsearch:

• GridPointData: The data plotted at a single gridpoint, e.g. smoothed training data, simulations
• PlotData: That data along with the experimental parameters defining the gridpoint
• SeriesData: The summary data to plot after a gridsearch
• Results: summary and gridpoint data, along with other parameters (e.g. metrics, gridsearch parameter values)

Data can be loaded with utils.load_results(hexstr), where hexstr is the XXXXX of the "results_XXXXXX.npy" files saved in a mitosis run. The data from multiple runs or across parameters can be combined with the following functions:

• plot_experiment_across_gridpoints: Plot only the training or test data of a single experiment across multiple parameters
• plot_point_across_experiments: Plot only the training or test data of multiple experiments for a single parameter set
• plot_summary_metric: Combine single subplots from the summary plot (e.g. F1 score across sim_params.t_end) across multiple experiments
• plot_summary_test_train: See how different parameter choices affect test/training data across multiple experiments

Because these are run after experiments, it's required to run complete gridsearch experiments before testing this stuff out. These experiments need to have plot_prefs set in order to store any intermediate plot data. The resulting files need to be opened and the hexstr extracted (since mitosis doesn't save this info in the database).

As a teaser, the functions can be called like:

from gen_experiments import utils

results = utils.load_results("580e49")
params = results["plot_data"][0]["params"]
utils.plot_point_across_experiments(
    params,
    ("Cubic Oscillator", "e3399c"),
    ("Duffing", "c8d97a"),
    ("Hopf", "8aa111"),
    ("Lotka-Volterra", "393bde"),
    ("Harmonic Oscillator", "46e3e1"),
    ("Van der Pol", "580e49"),
    ("Rossler", "2e7508"),
    style="training",
)

Also note that I blacked some of the existing code. Sorry, I should have done that in a different PR

[Jacob-Stevens-Haas]

FWIW, these are the experiments that I ran:

run_exps.sh

``` nohup python -m gen_experiments gridsearch --seed 19 \ --param ex_name=cubic_ho \ --param metrics=all \ --param other_params=4nonzero-cubic \ --param grid_params=duration-absnoise \ --param grid_vals=duration-absnoise \ --param grid_decisions=plot2 \ --param series_params=kalman-auto3 \ --param plot_prefs=test-absrel5 \ --param skinny_specs=duration-noise &> cubic_ho.log & nohup python -m gen_experiments gridsearch --seed 19 \ --param ex_name=sho \ --param metrics=all \ --param other_params=4nonzero-cubic \ --param grid_params=duration-absnoise \ --param grid_vals=duration-absnoise \ --param grid_decisions=plot2 \ --param series_params=kalman-auto3 \ --param plot_prefs=test-absrel5 \ --param skinny_specs=duration-noise &> sho.log & nohup python -m gen_experiments gridsearch --seed 19 \ --param ex_name=vdp \ --param metrics=all \ --param other_params=4nonzero-cubic \ --param grid_params=duration-absnoise \ --param grid_vals=duration-absnoise \ --param grid_decisions=plot2 \ --param series_params=kalman-auto3 \ --param plot_prefs=test-absrel5 \ --param skinny_specs=duration-noise &> vdp.log & nohup python -m gen_experiments gridsearch --seed 19 \ --param ex_name=lv \ --param metrics=all \ --param other_params=4nonzero-cubic \ --param grid_params=duration-absnoise \ --param grid_vals=duration-absnoise \ --param grid_decisions=plot2 \ --param series_params=kalman-auto3 \ --param plot_prefs=test-absrel5 \ --param skinny_specs=duration-noise &> lv.log & nohup python -m gen_experiments gridsearch --seed 19 \ --param ex_name=duff \ --param metrics=all \ --param other_params=4nonzero-cubic \ --param grid_params=duration-absnoise \ --param grid_vals=duration-absnoise \ --param grid_decisions=plot2 \ --param series_params=kalman-auto3 \ --param plot_prefs=test-absrel5 \ --param skinny_specs=duration-noise &> duff.log & nohup python -m gen_experiments gridsearch --seed 19 \ --param ex_name=hopf \ --param metrics=all \ --param other_params=hopf-cubic \ --param grid_params=duration-absnoise \ --param grid_vals=duration-absnoise \ --param grid_decisions=plot2 \ --param series_params=kalman-auto3 \ --param plot_prefs=test-absrel5 \ --param skinny_specs=duration-noise &> hopf.log & nohup python -m gen_experiments gridsearch --seed 19 \ --param ex_name=ross \ --param metrics=all \ --param other_params=lor-ross-cubic \ --param grid_params=duration-absnoise \ --param grid_vals=duration-absnoise \ --param grid_decisions=plot2 \ --param series_params=kalman-auto3 \ --param plot_prefs=test-absrel5 \ --param skinny_specs=duration-noise &> ross.log & ```