refactor algorithm.py and optimizer-classes

[bjuergens]

algorithm.py and the optimizer-classes are a huge mess. This started because we followed the design patterns used in DEAP. This was fine in the beginning, when we used most parts of DEAP "as intended" but as we added more features, that were incompatible with DEAP, the code become uglier and uglier.

This issue is not for a specific todo to improve the situation, and instead a place for discussing the problem.

To get this discussion started, I'd like to put some questions up for debate:

• What parts of algorithm.py are hard to read?
• What kind of optimizers may we want to use in the future?
• Should we get rid of IOptimizer entirely or do we just need modifications to this interface?

[bjuergens]

I also don't like the interaction between optimizer and episode-runner. Some parameters from the episode-runner affect the optimizers and vice versa. (e.g. fix_seed_for_generation, novelty, reuse_env)

I was just working on #39 and found no clean and easy way to pass execute the ep-runner with distinct sets of parameters. (I went for a quick & dirty way, and now I must go and take a shower).

A refactor should allow for a cleaner way to implement #39, than what I just did.

[bjuergens]

I do not like the handling of the seeds, how it is done at the moment. It adds a lot of complexity and it's hard to debug. The relevant code is here: https://github.com/neuroevolution-ai/NeuroEvolution-CTRNN_new/blob/23fedf85f9344c7bb6b926c4582cc6064e387c0a/neuro_evolution_ctrnn/tools/algorithms.py#L16

a simple solution would look like this:

• seed_for_generation is always initial_seed + generation. So the first generation uses seed initial_seed+1, the 2nd generation uses initial_seed+2 and so on
• seed_after_map is also set to initial_seed + generation.

note: the seed_after_map is there, because i noticed, that the random state in the main thread sometimes gets influenced by the random state in the worker-threads. So the random state needs to be resetted after the workers are done. (note: both optimizers use the random state to generate new individuals)

[bjuergens]

when https://github.com/neuroevolution-ai/NeuroEvolution-CTRNN_new/pull/87 is merged, I think we just need the following changes to have clean optimizer code:

• remove novelty search (re-implement clean, when needed again)
• remove toolbox (and move content to self)
• fix HOF-method (see issue #78)

ideas / follow up issues:

• create individual class which handles the stuff, that has been moved to the episoderunner in https://github.com/neuroevolution-ai/NeuroEvolution-CTRNN_new/pull/87
  • It is bad, that the eprunner not only runs episode, but instead has to update the individual. (violation of "single-responsibility principle). instead the eprunner should pass the fitness to a method in the individual class, and that method takes care of updating the individual
• refactor fitness_shaping. (currently it violates DRY because it repeats a lot of code for all fitnesses). Also it should inherit from deaps fitness, because that already give as what we need.
• new multiobjective target: reward individuals for having little variance in their genes (I believe that it will influence the training similarly to weight-decay)