search

heal-research / HeuristicLab

HeuristicLab - An environment for heuristic and evolutionary optimization
https://dev.heuristiclab.com
GNU General Public License v3.0
36 stars 16 forks source link

Create algorithm iterator optimizer #2431

Open HeuristicLab-Trac-Bot opened 9 years ago

HeuristicLab-Trac-Bot commented 9 years ago

Issue migrated from trac ticket # 2431

milestone: HeuristicLab 3.3.17 | component: Analysis | priority: medium

2015-07-16 11:47:01: @abeham created the issue

The algorithm iterator should execute an algorithm instance over and over until a certain stopping criterion is met. The stopping criteria are:

  • Maximum execution time reached
  • Maximum number of evaluated solutions
  • A certain target fitness was achieved

Design issues:

  • Algorithm Execution time is hard to obtain (similar problem with the introduction of terminators)
    • The precision of execution time from the algorithm is not good enough (~200ms)
  • Adding EvaluatedMoves and EvaluatedSolutions is complicated by different cost factors
    • Some algorithms/operators count evaluated solution-equivalents instead
    • Integer is sometimes too small to hold the amount of evaluated moves for long runs / big problems
  • Analyzers depend on BestAverageWorstQualityAnalyzer being run before them
    • Arguably, best-so-far quality and execution time are results that the algorithm itself should provide and not an analyzer
    • Quality progress chart should be calculated in a separate analyzer
  • ISingleObjectiveHeuristicOptimizationProblem specifies MaximizationParameter as an IParameter instead of an IValueParameter
  • Which parameters in a run characterize the algorithm instance?
    • E.g. Seed does not characterize the instance
    • Analyzers do not characterize the instance, but some "analyzers" may modify penalties
  • ResultCollection contains Result objects, but RunCollection.Results contains only the items
    • The Result objects are always in the way, they should be removed
HeuristicLab-Trac-Bot commented 9 years ago

2015-07-16 11:50:16: @abeham changed status from new to accepted

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-16 11:50:16: @abeham commented

r12764:

  • branched relevant plugins
  • added analyzer to calculate quality vs evaluation first-hit graph
  • (also added results parameter)
HeuristicLab-Trac-Bot commented 9 years ago

2015-07-16 11:51:08: @abeham commented

r12766: forgot to commit

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-17 09:26:27: @abeham commented

r12771:

  • Added run collection view
  • Changed name of analyzers
  • Modified algorithms to include Execution Time as a result

Logscaling is still an issue

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-17 09:46:06: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-17 12:34:58: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-17 12:34:58: @abeham commented

r12774: minor improvements

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-23 20:27:47: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-23 20:27:47: @abeham changed title from Algorithm performance comparison by ECDF to Run-length distribution (RLD) Analysis

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-24 15:32:02: @abeham commented

r12803: worked on RLD analysis

  • started implementation of IRRRun
  • renamed view from ECDF to RLD
  • reverted algorithms (execution time)
  • changed per clock analyzer
HeuristicLab-Trac-Bot commented 9 years ago

2015-07-26 21:34:51: @abeham commented

r12804:

  • worked on IRRestarter (early abort still troublesome)
  • Updated RLD view to allow defining targets
  • Attempting to handle maximization/minimization

Fun detail:

  • HeuristicLab.Analysis references HeuristicLab.Optimization
  • HeuristicLab.Optimization.Views references HeuristicLab.Analysis.Views

IRRestarter is currently placed in HeuristicLab.Analysis, its view in HeuristicLab.Optimization.Views.

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-26 21:36:36: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-27 12:50:06: @abeham commented

r12805: fixed error due to removed resx file

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-27 16:10:32: @abeham commented

r12806: added options to add results based on targets or budgets

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-28 14:41:18: @abeham commented

r12808:

  • added ability to plot curves for multiple targets at once
  • fixed bug in IndexedDataTableView regarding log-scaling
  • added result that counts restarts
  • fixed bugs in IRRestarter
  • Set LineWidth = 2 in analyzers for chart
  • fixed bugs in RLD view
HeuristicLab-Trac-Bot commented 9 years ago

2015-07-28 14:44:34: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 9 years ago

2015-07-30 15:27:11: @abeham commented

r12813:

  • Added calculation of RTs, RTus, FEs, FEus and prune runs in alg before every restart
  • merged changes from trunk
HeuristicLab-Trac-Bot commented 9 years ago

2015-07-31 08:36:50: @abeham commented

r12822: if multiple targets are plotted, the same color is used for each group

HeuristicLab-Trac-Bot commented 9 years ago

2015-08-02 01:15:32: @abeham commented

r12825:

  • removed output of RTs, RTus, ...
  • added 2nd row to charts that state restarts until a target was achieved
  • added option to include or exclude solution results
HeuristicLab-Trac-Bot commented 9 years ago

2015-08-02 10:57:04: @abeham commented

r12826: omitting empty series

HeuristicLab-Trac-Bot commented 9 years ago

2015-08-02 16:24:36: @abeham commented

r12831: updated to trunk

HeuristicLab-Trac-Bot commented 9 years ago

2015-08-03 23:33:33: @abeham commented

r12834: Changed analyzer to always output last value (even if quality is equal)

HeuristicLab-Trac-Bot commented 9 years ago

2015-08-05 00:07:01: @abeham commented

r12838: updated RLD analysis view

  • Added analysis by cost
  • Prepared tab for output of ERT tables
HeuristicLab-Trac-Bot commented 9 years ago

2015-08-10 12:49:54: @abeham commented

r12841:

  • Added calculation of ERT results tables
  • Added grouping by problem to avoid having different problems in one graph
  • Some restructuring to avoid duplicate code
HeuristicLab-Trac-Bot commented 9 years ago

2015-08-13 14:19:36: @abeham commented

r12856: Added AlgorithmIterator instead of IRRestarter

  • AlgorithmIterator implements IAlgorithm whereas IRRestarter implemented only IOptimizer
  • Deriving from Algorithm leads to better design
HeuristicLab-Trac-Bot commented 9 years ago

2015-08-13 16:04:49: @abeham commented

r12859: Renamed AlgorithmIterator to IteratedAlgorithm

HeuristicLab-Trac-Bot commented 9 years ago

2015-08-13 17:12:07: @abeham commented

r12862: modified EngineAlgorithm to clear log on prepare

HeuristicLab-Trac-Bot commented 9 years ago

2015-08-13 17:22:23: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 9 years ago

2015-08-16 00:53:45: @abeham commented

r12864:

  • fixed bug in iterated algorithm
  • changed RLD view to use % deviation as targets
  • RLD view now always groups by problem
HeuristicLab-Trac-Bot commented 9 years ago

2015-08-17 09:32:08: @abeham commented

r12865:

  • Added label on last data point
  • Changed target in cost view to display as ratio to best-known or best found
HeuristicLab-Trac-Bot commented 9 years ago

2015-08-20 08:56:25: @abeham commented

r12888:

  • Some bug fixes
HeuristicLab-Trac-Bot commented 9 years ago

2015-08-21 22:19:08: @abeham commented

r12890: fixed a bug in the iterated algorithm

HeuristicLab-Trac-Bot commented 9 years ago

2015-09-04 09:20:47: @abeham commented

r12939: fixed calculation of ERT table when a certain group doesn't contain any runs of a certain problem

HeuristicLab-Trac-Bot commented 9 years ago

2015-09-19 21:20:07: @abeham commented

r12956: Extracted class that calculates ERT values from runs

HeuristicLab-Trac-Bot commented 8 years ago

2015-11-18 13:32:24: @Shabbafru changed milestone from HeuristicLab 3.3.13 to HeuristicLab 4.0

HeuristicLab-Trac-Bot commented 8 years ago

2015-11-26 09:34:06: @abeham changed milestone from HeuristicLab 4.0 to HeuristicLab 3.3.14

HeuristicLab-Trac-Bot commented 8 years ago

2015-12-16 17:05:23: @abeham commented

r13475: updated from trunk, worked on okb connection for downloading knowledge base

HeuristicLab-Trac-Bot commented 8 years ago

2016-03-26 11:10:10: @abeham commented

r13736: fixed analysis for runs that terminated earlier than others

HeuristicLab-Trac-Bot commented 8 years ago

2016-03-26 11:14:20: @abeham commented

r13737:

  • removed IRRestarter (replaced by IteratedAlgorithm)
  • removed cut-off estimation view (didn't work satisfactorily)
HeuristicLab-Trac-Bot commented 8 years ago

2016-03-26 11:20:42: @abeham commented

r13738: fixed bug in IteratedAlgorithm (exception when problem is loaded, but no algorithm set)

HeuristicLab-Trac-Bot commented 8 years ago

2016-07-11 14:32:48: @abeham commented

r14035:

  • Fixed bug that occurs when runs of unequal length are aggregated over multiple problem instances
HeuristicLab-Trac-Bot commented 8 years ago

2016-07-13 10:52:07: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 8 years ago

2016-07-13 10:52:07: @abeham commented

r14058: Refactored ResultsParameter

HeuristicLab-Trac-Bot commented 8 years ago

2016-07-13 12:12:12: @abeham commented

r14059: Removed NonDiscoverableType attribute on ResultParameter

HeuristicLab-Trac-Bot commented 8 years ago

2016-07-14 15:46:51: @abeham commented

r14070: fixed some bugs in the result parameter view (thx to mkommend)

HeuristicLab-Trac-Bot commented 8 years ago

2016-07-14 15:50:27: @abeham edited the issue description

HeuristicLab-Trac-Bot commented 8 years ago

2016-07-14 15:50:27: @abeham changed title from Run-length distribution (RLD) Analysis to Create algorithm iterator optimizer

HeuristicLab-Trac-Bot commented 8 years ago

2016-07-18 17:06:13: @abeham changed milestone from HeuristicLab 3.3.14 to HeuristicLab 3.3.15

HeuristicLab-Trac-Bot commented 7 years ago

2017-06-24 23:00:51: @abeham changed milestone from HeuristicLab 3.3.15 to HeuristicLab 3.3.16

HeuristicLab-Trac-Bot commented 5 years ago

2019-04-12 13:52:42: @abeham changed milestone from HeuristicLab 3.3.16 to HeuristicLab 3.3.17