search

LearnersGuild / game-prototype

Lightweight, minimal implementation of game mechanics for rapid experimentation and prototyping.
0 stars 0 forks source link

Erase pro players from Elo history #95

Closed shereefb closed 7 years ago

shereefb commented 7 years ago

Strategic Goals this issues impacts

What are the benefits of this change, and whom do they impact?

Describe the change, and provide any needed context.

Changes

We'll keep running Elo but only amongst the learners. Because their skill differentials are significantly smaller, Elo should calibrate more effectively within that pool.

Once we have enough data about how learners' Elo evolve and how well it matches to our Shared Perception of Reality®, then we can try at a better guess of what pro player's Elo should be and re-introduce them into the Elo pool.

Context

For the teams in which pro players were involved as an "advanced player", they were not able to play to their full potential because of the massive skill differential. Because of this, their Elo is artificially low, which means that when and if they do play to their full potential, the consequences for their team's Elo will be unfairly harsh.

The truth is we don't know what the Elo of a professional player should be. They are too far ahead of the rest of the pack for us to make a decent guess. So we should first have Elo calibrate to the learners themselves, try and make the rating as accurate as possible for their skill distribution, and then figure out how to introduce much more advanced/pro players.

tannerwelsh commented 7 years ago

ready for review @LearnersGuild/software

bundacia commented 7 years ago

LGTM. I liked the idea that players could compare their stats to a professional player. Hopefully we'll be able to add them back eventually or find some other way to allow players to measure themselves against someone they know is successful in the field. I suppose once we have graduates we'll be able to give learners the average stats of someone who got a job upon graduation, and that will reduce the need for us to have pro players with stats the learners can compare themselves against.

jeffreywescott commented 7 years ago

One thing to flag for implementation is to figure out how we'll know when someone is a pro player. Is it a role?

tannerwelsh commented 7 years ago

how we'll know when someone is a pro player. Is it a role?

Could be. Could also be a property of the project, something like ignoreElo: [<playerId>].

bundacia commented 7 years ago

@tannerwelsh: We'd still need a way to identify pro players so we know which playerIds to stick in that ignoreElo list. This probably needs to be a flag in the player or user someplace.

heyheyjp commented 7 years ago

@shereefb: I like the idea in general, but I have a small issue with this being strictly limited to omission of pro players from Elo rating calculation. Based on my experiences on the floor, it wasn't just pro players who had this problem...multiple pod 1 people felt they needed to similarly "dumb down" their play. Should we attempt to somehow account for this, too?

shereefb commented 7 years ago

@prattsj this game only works if it doesn't require "dumbing down". As a player, I should be in my ZPD at all times, and my stats should reflect my ability to do that.

So, for example, Devon starting "dumbing down" for the first 3 weeks of the second cohort. His stats should start hurting, and it should force him to play in a way where he is in his ZPD. Devon's Elo dropping is one way to signal that he should start playing in full force. So, no, don't think we should account for that at all. Taking Devon's ELO out of the equation when he's an advanced player would remove important feedback to him.

tannerwelsh commented 7 years ago

@prattsj it's a worthy point, and I agree with @shereef that stats should reflect it.

IMHO, this issue is more about a difference of degree than a difference of kind. Whenever working across a skill differential, there are going to be times where more advanced players may have to go slower or use less technical language than they otherwise would.

But I'd hope that this happens far less often, and to a less severe extent, when the experience differential is measured in weeks and not years, or even decades.

shereefb commented 7 years ago

lgtm

@jeffreywescott please consider moving this to top of the backlog

tannerwelsh commented 7 years ago

Just updated in game-prototype, and changed the stats spreadsheets to match new Elo calculation (for player-facing stats, I only updated sheets for cycle 10+)

jeffreywescott commented 7 years ago

Issue moved to LearnersGuild/game #572 via ZenHub