Jacob-Stevens-Haas / gen-experiments

Pysindy experiments. Composable and extensible
MIT License
1 stars 2 forks source link

Allow gridsearching on the min of some indices, and also plot the optimum #10

Closed Jacob-Stevens-Haas closed 10 months ago

Jacob-Stevens-Haas commented 11 months ago

Sorry Yash, this is a twofer. I added it as a draft PR, so no review needed needed until I mark it ready for review. Just read and understand this. Ask questions about anything, like types, that you could use clarity on.

  1. The code in gridsearch.py finds the max along some parameters for each grid point of other parameters, e.g. "Find the optimal smoothing distance in a Savitsky-Golay smoother that maximizes F1 score". The code was only written to handle the max, not the min, which means that it's also gridsearching, for example, the max mean squared error. YIKES, we want to be able to control whether we maximize or minimize some metrics.
  2. Currently, the plot_prefs argument in gridsearch.run lets you choose which parameters in the gridsearch get their own plots. This means that we can't directly compare the individual plots that come out, because Kalman includes its own parameter identification methods, but other methods require specifying parameters BEFORE gridsearching to find the optimal parameters. This PR moves plotting the individual results until AFTER gridsearching and allows specifying the "optimal" parameter in a PlotPrefs.
Jacob-Stevens-Haas commented 11 months ago

Ok it's good to go at this point. Changes to the calling signature are just the inclusion of an indexing tuple to locate a gridsearch result. Existing experimental code should still work, e.g.

nohup python -m gen_experiments gridsearch --seed 19 \
    --param ex_name=sho \
    --param metrics=all \
    --param other_params=4nonzero-cubic \
    --param grid_params=duration-absnoise \
    --param grid_vals=duration-absnoise \
    --param grid_decisions=plot2 \
    --param series_params=kalman-auto3 \
    --param plot_prefs=test-absrel5 \
    --param skinny_specs=duration-noise &> sho.log &