Run EA experiments and tests over enemies, instances, repeats

UPDATES: This addresses issue #13 . It's still a work in progress. This has already incorporated PR #14 .

The main idea is that each EA instance we want, we create a new class based on evo_utils.BaseEAInstance.


class BaseEAInstance(ABC):
    """Base class for EA instances.
    """

    def __init__(self, experiment_directory='experiments/tmp'):
        self.experiment_directory = experiment_directory
        self.enemies = None
        self.player_controller = None
        self.toolbox = None
        self.population = None
        self.hall_of_fame = None
        self.stats = None
        self.final_population = None
        self.logbook = None
        self.best_individuals = None
        self.top_scores = None

        # Set directory for saving logs and experiment states
        if not os.path.exists(self.experiment_directory):
            os.makedirs(self.experiment_directory)

    def evolve(self):
        raise NotImplementedError()

    def __repr__(self):
        params = {
            'enemies': self.enemies,
            'toolbox': self.toolbox.__repr__(),
            'pop_size': len(self.population),
            'top_fitness': np.max(self.top_scores),
        }
        return f"EA instance: {params}"

Then, experiments_evolve.py loops through our different EA instance classes and runs them for N repeats on M different enemies, saving the results to saved_instances/<timestamp>_best_genomes.json.

Pseudocode:

for ea_instance in [ea_config_1, ea_config_2]:
    for enemy in [1, 3, 5]:
        for repeat in range(10):
            # Evolve for N generations
            # Save best individual

From there, we load those best genomes into experiments_test.py, which does robust testing and generates a dataframe of results (work in progress).

Pseudocode:

for enemy in [1, 3, 5]:
    for individual in best_individuals[enemy]:
        for repeat in range(5):
            # Test individual and save results

gianlucatruda / evocomp

Run EA experiments and tests over enemies, instances, repeats #15