EbTech / Elo-MMR

Skill estimation systems for multiplayer competitions
https://arxiv.org/abs/2101.00400
MIT License
174 stars 21 forks source link

Last position people are awarded positive rating changes. #23

Open atiabjobayer opened 11 months ago

atiabjobayer commented 11 months ago

Used Algorithm: mmr

Consider a situation where 3 user participated in a contest[user1, user2, user3]. Consider it's a programming contest. Let's say their scores are [300/500, 200/500, 0/500].

Now the basic sense suggest that since the user3 scored absolutely zero, he should get a negative rating change. But what happens is, if user1 and user2 have higher rating than user3 before the contest, then user3 receives a positive rating change, even if he scores zero.

Is there any way to prevent this behavior? Doesn't seem right at all.

EbTech commented 11 months ago

Hmm it's hard to tell what exactly happened without going through the math in detail, but this shouldn't usually happen. From the 0/500 player's perspective, the algorithm treats the result as 1 loss against the 300 player, 1 loss against the 200 player, and either 1 loss + 1 win or 0.5 loss + 0.5 win (depending on the split_ties parameter) against themself. Losses never help, so you might like to try removing one or both of the winners from the match to confirm that having fewer losses results in a higher rating for the 0/500 player.

That leaves the question of why a player's rating would increase from a neutral result, where they play only against themself. The answer is that probably mu didn't change, but sigma decreased; if we publish something like mu - 2*sigma, then the published rating will be higher. sigma changes a lot for new players, but stabilizes later so you'd no longer see this effect. Of course, a 1-player match is completely uninformative so sigma really shouldn't be updated, but Elo-MMR's measure of information gain is based on the limit as the number of participants goes to infinity.