Optimizer should really compare best candidate model predictions to observations.

[russelljjarvis]

Optimizer should really compare best candidate model predictions to observations, as well as comparing results to an exhaustive search.

To achieve this I am running:

preds = [ t.generate_prediction(model) for t in tests ]

which results in more invocations of the simulator.

It would be more efficient to store the last predictions in the test object, in the same way that observations are stored.

As a concrete example:

in the tests/__init__.py file:

generate_prediction acts on a local variable that is returned by calling the function.

If instead the test classes had a variable declared like:

self.stored_prediction = None: which is subsequently over written inside generate_prediction with something like:

prediction = {'mean':np.mean(threshes) if len(threshes) else None,
               'std':np.std(threshes) if len(threshes) else None,
               'n':len(threshes)}
self.stored_prediction = prediction
return prediction

then I can iterate over completed tests and plot the the differences between observations and predictions, as would be a faster and more reliable way to debug the evolution of the error in the GA.

I have also included the standard definition of APThresholdTest class including its generate_prediction method for contrast.

class APThresholdTest(VmTest):
    """Tests the full widths of action potentials at their half-maximum."""

    required_capabilities = (cap.ProducesActionPotentials,)

    name = "AP threshold test"

    description = ("A test of the membrane potential threshold at which "
                   "action potentials are produced.")

    score_type = scores.ZScore

    units = pq.mV

    ephysprop_name = 'Spike Threshold'

    def generate_prediction(self, model):
        """Implementation of sciunit.Test.generate_prediction."""
        # Method implementation guaranteed by
        # ProducesActionPotentials capability.
        model.rerun = True
        threshes = model.get_AP_thresholds()
        # Put prediction in a form that compute_score() can use.
        prediction = {'mean':np.mean(threshes) if len(threshes) else None,
                      'std':np.std(threshes) if len(threshes) else None,
                      'n':len(threshes)}
        return prediction

[rgerkin]

Implemented in russelljjarvis/dev branch. Russell, could you reference a block of code or commit number here and close the issue?

[russelljjarvis]

Sorry Rick I didn't see this. However, as we have decided to remove Test Class attributes prediction and the same or similar functionality can be achieved by pulling observations/predictions out of score, and storing them in lists.