search

ZeusWPI / aichallenge

A bot writing contest
https://zeus.ugent.be/bottlebats
9 stars 6 forks source link

Match only bots with comparable elo #97

Open wschella opened 8 years ago

iasoon commented 8 years ago

Why do you want to do this? Isn't cancelling out differences in skill the point of elo rating?

wschella commented 8 years ago

Yes, but not to the level of matching the best bot vs. the worst, and certainly not if this happens multiple times (due to small playerbase). There simply is no gain for the best players to play the worst, since the risk off losing due to unbalanced maps is too high.

iasoon commented 8 years ago

Unbalanced maps are an unrelated problem that should either be solved by only generating balanced maps, or playing a second match with swapped positions. I'm in favour of this second solution because it allows you to draw.

wschella commented 8 years ago

That doesn't solve the problem that there is little incentive for very high elo bots to play very low elo bots since there is very little to win. Agreed, chances to win are high, but this results in countless pointless matches with the sporadic upset. I'd like to gradually migrate to less, but more important matches, and matching bots with (somewhat) comparable skill is a good step in that direction.

iasoon commented 8 years ago

There is little incentive for high elo bots to play low elo bots, but there is much incentive for low elo bots to play high elo bots.

We obviously want less matches. However, I think there is more cpu time to gain in not playing useless matches - matches that have already been played enough, of which the outcome is entirely predictable. This is just as true for bots that are close in elo.

A solution might be to limit the amount of battles there can be between two bots. There will be more matches each time a new bot enters the system, that will still allow you to adjust the ratings.

Every while the match counters could be reset (or lowered) to keep the ratings from deviating from an equilibrium too much.

wschella commented 8 years ago

Is there high incentive tho? It is no fun to keep losing consistently against bots above your skill level. I'm mostly drawing comparisons to ELO-based online and board games; the difference is that your bot won't learn from playing against a higher ranked one.

My main reason for less matches is not CPU-time, but more so to increase a coders connection to his bot. The bots matches should be exiting like sport games, unbalanced matches will be mostly onesided and boring.

iasoon commented 8 years ago

Yes, because someone having a higher rating than you does not mean he is better than you.

Due to complex maps and countering strategies, it might well occur that A beats B, B beats C, but C beats A. If C never gets to fight A, he will be at the bottom of the leaderboard, while these bots should be ranked equally.

wschella commented 8 years ago

This line of thinking is a slippery slope. On a larger scale, if C specifically counters A, but performs badly against most others, A's score will be severely impacted due to C's low elo, while C is actually a bad bot, because it performs bad in general.

To completely exaggerate: you could nominate multiple bots specifically countering your eternal nemesis to decrease his elo dramatically and putting you in first place.

iasoon commented 8 years ago

If A gets beaten by a stupid bot like that, he definitely shouldn't be ranked that highly.

Being good at a game means being flexible enough to win in a variety of circumstances.

wschella commented 8 years ago

You prove my point. Being a good at a game means being flexible enought to win in a variety of circumstances. This means if you can't break out of the situation described in your previous comment, you don't deserve to rank higher. Matching only comparable elos should provide enough opportunity to climb the rankings.

PS: That bot doesn't have to be stupid, just extremely focused on countering 1 bot's strategy.