efficacy of parameter recovery using current optimization routine

[rythorpe]

While working on #446, I discovered that the current method for optimizing evoked drive parameters fails to recover the mean drive time (mu) of a single proximal drive that has been offset from its original value. This is particularly troublesome given that optimization of evoked drive timing is one of the simplest and most common use-cases that one might implement when attempting to use the optimization module to infer causal mechanisms underlying an experimentally measured evoked response. We should explore this further, and at least attempt to rule out the possibility that

1. I've made a stupid mistake when testing this, and
2. seemingly arbitrary implementation choices (e.g., our use of scipy.optimize.fmin_cobyla and the default arguments we pass into it within the hnn_core.optimization module) are impairing optimization efficacy.

See here for two demo scripts, one that runs on the current master branch at 23f9a68 and one that runs with the updated API in #446.

[ntolley]

Very curious, is the RMSE reasonably small and the optimization is finding a "valid" solution according to our definition? Or is it completely failing to fit the curve.

The potential 3rd option is that the minimum of the optimization objective function does not uniquely specify parameters.

[rythorpe]

The RMSE is indeed fairly small (on the order of 1e-3 nAm), however, I've played around a bit with fmin_cobyla's rhoend parameter and it doesn't seem to help. I've limited the parameter space down to 10-dimensions (mu, sigma, 4 ampa weights, 4 nmda weights) and would expect the optimization to at least move mu a little towards the ground-truth mu-value that generated the target dipole, but the algorithm consistently moves in the opposite direction and spends way too much time exploring the other parameters that should remain constant.

[jasmainak]

can you plot the objective function (weighted rmse?) with x-axis as iteration number? COBYLA is gradient-free, so it should find the "true" parameters eventually ...

[rythorpe]

The weighted RMSE definitely tends to decrease, albeit in a very inefficient and roundabout manner. COBYLA seems to find an obscure combination of weight and sigma parameters that decreases the weighted RMSE by a very small amount. The fact that it struggles to recover mu makes me wonder if the time-dependent weighted RMSE is the problem.

[jasmainak]

A plot would be helpful because you can actually spot things, e.g., if it's stuck without improving for a while. If the objective function decreases fine then we can rule out problems with the solver.

[rythorpe]

I probably won't be able to get to it this week, but will try to follow up with a plot soon!

[rythorpe]

Adding a note that once this issue is resolved, we should update the optimization tests as suggested here and here by @ntolley.

[ntolley]

Also noting @rythorpe's suggestion to consider using optimization libraries that enforce hard boundaries to avoid hard coding limits on parameter values

[rythorpe]

I believe the current optimization routine has moved past this.