search

aturfah / cmplxsys530-final

Final Project for Complex Systems 530 WN18
MIT License
2 stars 2 forks source link

Investigate why PlanningFloatzel plays so many games #140

Closed aturfah closed 5 years ago

aturfah commented 5 years ago

Using the sample Pokemon simulation (files included below), notice how PlanningFloatzel plays significantly more games against itself vs other agents who play noticeably less games against themselves (2k vs 1k).

This could be due to the fact that the PlanningFloatzel agents are close enough in Elo that they mainly play one another, but this doesn't explain why the PlanningCBSpinda doesn't play a similar amount of games vs itself.

elo matchups

To reproduce

sample_pokemon.yaml

game_choice: 4
num_players: 60
num_games: 15000
multithread: 1
config: "sample_simulations/sim_configs/sample_pkmn_config.json"

sample_pkmn_config.json

[
    {
        "proportion": 0.15,
        "team_file": "simulation_floatzel.txt",
        "agent_type": "RandomFloatzel",
        "agent_class": "basic"
    },
    {
        "proportion": 0.15,
        "team_file": "simulation_ivysaur.txt",
        "agent_type": "RandomIvysaur",
        "agent_class": "basic"
    },
    {
        "proportion": 0.15,
        "team_file": "simulation_spinda.txt",
        "agent_type": "RandomSpinda",
        "agent_class": "basic"
    },
    {
        "proportion": 0.15,
        "team_file": "simulation_floatzel.txt",
        "agent_type": "PlannningFloatzel",
        "agent_class": "basicplanning",
        "agent_tier": "pu"
    },
    {
        "proportion": 0.15,
        "team_file": "simulation_ivysaur.txt",
        "agent_type": "PlanningIvysaur",
        "agent_class": "basicplanning",
        "agent_tier": "pu"
    },
    {
        "proportion": 0.15,
        "team_file": "simulation_spinda.txt",
        "agent_type": "PlanningSpinda",
        "agent_class": "basicplanning",
        "agent_tier": "pu"
    },
    {
        "proportion": 0.1,
        "team_file": "cb_spinda.txt",
        "agent_type": "PlanningCBSpinda",
        "agent_class": "basicplanning",
        "agent_tier": "pu"
    }
]
aturfah commented 5 years ago

Ran with same config but a random ladder (add ladder: 1 to sample_pokemon.yaml) and got results below. (Note there are less PlanningCBSpindas than other agents so less numbers is expected).

matchups

The results above may be a bias of the pairing ladder's matching function (and may go away with #135), but will need to investigate.

aturfah commented 5 years ago

Ran simulation with 150000 games, 600 players. same proportions. Player1.type is the column if interest because it is the randomly chosen one.

Totals by type:

PlanningCBSpinda is ~2/3 of everyone else, which makes sense as they are only 10% of the total population whereas everyone else is 15%. All the other numbers are within range of each other.

This is NOT a bug. Attaching outputs (as .txt)

PKMNTypes_2019-02-02_15-42-22-880227.txt PKMNPlayers_2019-02-02_15-42-22-837813.txt

aturfah commented 5 years ago

Changed label to be more appropriate to the nature of this issue.