Choose Hyperparameters for Different Experiments

[philswatton]

The genetic algorithm has a large number of hyperparameters which need to be chosen for the hypotheses in #19:

Operations:

• Crossover method/whether to crossover
• Crossover probability
• How many pokemon to crossover (can be random each time)
• Whether to mutate
• Whether to mutate inverse to fitness instead of using a simple probability
• Mutation probability
• How many pokemon to mutate (can be random each time)

Battle/fitness:

• fitness function to use (currently standardised win ratio vs BT model)
• match-up function (i.e. all vs all vs graph-based sampling)
• N of teams
• N of matches per team (if using graph-based sampling)
• N of battles per match

General:

• N of generations

---

Some considerations:

• The role of crossover is to swap good 'genes' between chromosomes
• Crossover probability is typically useful to keep high. In our context, you can also think of it as determining how many times a team needs to be selected before you'd expect it to be kept in the next population unaltered (e.g. a crossover prob of 50% would mean you need to select a team twice before you'd expect it to appear in the next generation unaltered, ignoring mutation probability)
  • Too high, you miss out on potentially syngergystic pairings. Too low, you fail to properly explore the space
• The role of mutation is to maintain genetic diversity. Too high, you start losing good genes too often. Too low, you can accidentally remove important sources of diversity early without a way to add them back in
• Some 'typical' values I found online were 0.95 for crossover probability and 0.01 for mutation probability

---

My memory of where discussions got to is:

1v1

• Selection as normal via fitness
• 'Crossover' is meaningless here, should be turned off
• Mutation may be important, especially as if Pokemon are removed early (and just got unlucky), this is the only route back in
• But: we may want to focus on mutating worse pokemon out, so use mutation inverse to fitness

2v2

• Selection as normal via fitness
• 'Crossover' has a role here, as teams can do well for two reasons:
  • One of the pokemon is especially good (we want to crossover)
  • Two pokemon synergise well (we wouldn't want to crossover)
• We'd want a lower crossover probability than 6v6, but not a low one either because of this. Could start with 0.75
• Mutation prob of 0.01 to start with

6v6

• Selection as normal via fitness
• Crossover is likely v important here for searching the space
• Could start with 0.95 (we could try and be more nuanced, but I don't have a strong take)
• Mutation prob of 0.01 to start with

General

• If we want a large N of teams, then BT + graph sampling of matches will be important
• Genetic Algo isn't necessary for 1v1, but 1v1 does offer a good test case for convergence towards the tiers
• I would prioritise unique matchups over repeating matches, as with enough matches losses through bad luck shouldn't matter too much while new matchups offer more information

---

All in all, the goal here is to work out some initial arguments, and possibly to suggest we write these up into a set of config files (YAML or JSON) with a script able to take a config path argument. We should discuss possible hyperparams more

[philswatton]

I don't have my notes with me from discussions during the week today, but I'll dig them up at some point and possibly edit this issue against them