testing: verify that statistical measures other than mean (np.mean) work within Evaluator.make_statistic_evaluator

[tjlca]

opt_parameters['statistic']= 'np.mean'

    if opt_parameters['statistic'] == "np.mean":
            opt_parameters['statistic'] = np.mean

[rjhanes]

When optimizing, users can theoretically define any statistical measures implemented in numpy as the measure (of the objective function value) on which to optimize. The default is np.mean. We haven't tested alternate statistical measures lately (to my knowledge) and so I'll be verifying that measures like median and percentiles also work within the optimization.

Testing plan:

• [x] np.median
• [x] np.percentile at 10th
• [x] np.percentile at 90th

This covers virtually all expected use cases.

If testing fails, then the make_statistic_evaluator method within the Evaluator class will need to be updated to ingest any failed measures.

Bonus: Passing around function names as parameters isn't good practice from a security standpoint. If time allows, update make_statistic_evaluator to ingest strings instead of callables and convert those strings to the relevant functions using if/then or similar statements.

Timeline: Complete by or on Friday, October 27.

[rjhanes]

Testing performed in pv_residential_simple. All tested statistics performed as expected.

To evaluate percentiles or quantiles, this is the syntax:

statistic    = lambda x: np.percentile(x, 0.9)

I'll start adding string-to-function processing in make_statistic_evaluator for the reasons stated in the previous comment, and also because the necessary syntax for evaluating percentiles won't necessarily be obvious to new Tyche users.

[rjhanes]

Evaluator methods (make_statistic_evaluator and a few others in the Evaluator class) have now been updated to use string-format parameters to define statistical functions. Function options are defined in a dict under the Evaluator init method, and currently are as follows:

    # dictionary of statistical function options 
    # for use in the various evaluator methods
    # add entries to this dict to expand the options
    self.statistic_lookup = {
      'mean': np.mean,
      'median': np.median,
      'standard deviation': np.std,
      '10th percentile': lambda x: np.percentile(x, 0.1),
      '25th percentile': lambda x: np.percentile(x, 0.25),
      '75th percentile': lambda x: np.percentile(x, 0.75),
      '90th percentile': lambda x: np.percentile(x, 0.9)
    }  

To use any method with a statistic parameter, the syntax is now statistic = 'mean' or statistic = '10th percentile' or similar.

Heads up for @tjlca - this will have implications for your GUI backend code, but hopefully will simplify things overall.

[rjhanes]

Resolved in PR #160