ENH: Additional KPI options and caching

[flongford]

Description

Includes additional attributes on KPISpecification class and implements caching of raw KPI values in the BaseOptimizerEngine class.

Related Issues

293

Summary

KPISpecification class now contains 4 new attributes:

• target_value: A value to offset the minimisation routine if objective == 'TARGET'
• use_bounds: Whether to include upper and lower bounds on the KPI values during the optimisation
• lower_bound: A lower bound for the KPI value
• upper_bound: An upper bound for the KPI value

BaseOptimizerEngine class contains 3 new methods:

• cache_result: stores MCO parameter values and corresponding evaluated KPIs
• retrieve_result: retrieves KPIs for a given set of MCO parameter values
• _get_kpi_cache_key: generates a hashable object from a set of MCO parameter values

Additionally, we enhance the BaseOptimizerEngine._score function to both cache the raw KPI values returned from the evaluator, and then return the minimisation score, based on each KPISpecification.objective value.

The IOptimize.optimize_function and BaseOptimizerEngine.optimise method have been slightly amended to expect keyword arguments, which is expected to be useful for passing in any additional values during the MCO.

Notes

• All spellings of "optimise", "minimise" and "maximise" in the code base have now been converted to American English

Future Work

• Other FORCE repos (force_wfmanager, force_nevergrad) should be updated to make use of these new features
• Some testing objects have been changed to accommodate new API and could be tidied up / moved to probe / dummy unit test modules.

Changes

• New attributes in KPISpecification class
• KPI value caching in BaseOptimizerEngine._score method
• convert_to_mask renamed to convert_to_score and can now handle "TARGET" objctives

[codecov[bot]]

Codecov Report

Merging #353 into master will increase coverage by 0.05%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #353      +/-   ##
==========================================
+ Coverage   97.42%   97.47%   +0.05%     
==========================================
  Files          64       64              
  Lines        1783     1821      +38     
  Branches      192      202      +10     
==========================================
+ Hits         1737     1775      +38     
  Misses         37       37              
  Partials        9        9              

Impacted Files	Coverage Δ
force_bdss/mco/optimizers/i_optimizer.py	100.00% <ø> (ø)
force_bdss/mco/optimizers/scipy_optimizer.py	73.84% <ø> (ø)
force_bdss/core/kpi_specification.py	100.00% <100.00%> (ø)
.../optimizer_engines/aposteriori_optimizer_engine.py	87.50% <100.00%> (+0.83%)	:arrow_up:
...dss/mco/optimizer_engines/base_optimizer_engine.py	100.00% <100.00%> (ø)
force_bdss/mco/optimizer_engines/utilities.py	100.00% <100.00%> (ø)
...mco/optimizer_engines/weighted_optimizer_engine.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0141489...471115e. Read the comment docs.

[flongford]

I think I get it (see comments below). I guess the nevergrad plugin should still work without any change. Have you tested it?

So running the unit tests only results in some failures because of changing MINIMISE to MINIMIZE etc. Everything else appears to run ok.

[flongford]

I think I get it (see comments below). I guess the nevergrad plugin should still work without any change. Have you tested it?

So running the unit tests only results in some failures because of changing MINIMISE to MINIMIZE etc. Everything else appears to run ok.

Euurrrggghhh... I just realised EVERY example workflow JSON will be broken by this spelling change - I'm going to remove it for now any make a future issue to implement it.

[flongford]

@sparsonslab - merging this now should not break anything in the enthought_example, force_nevergrad and force_wfmanager repos.

One unit test needs to be changed in the ITWM repo simply because we reduce the number of times the Workflow.evaluate function needs to be called by caching the results: https://github.com/force-h2020/force-bdss-plugin-itwm-example/pull/84