shereefb
commented
8 years ago @prattsj , @jeffreywescott , @bundacia heads up.
One of my main take aways from introducing stats to learners, is how "God stats" like XP that attempt to roll up many sub-stats maybe less useful than I previously thought they would be. XP is trying to do too much, and we're using it for at least two different purposes.
I've been thinking since Friday about using a rating/ranking algorithm along with XP, and wanted to capture my thoughts in a game design issue.
Wanted to give you guys an early heads up about this so that you can weigh in early and frequently as this comes down the pipelines.
Next step for me is to discuss with @tannerwelsh and if he's up for it, start to run some ELO ranking simulations with the current data set we have and see wether or not it gives us a more accurate/higher resolution picture of our learners as they stand (by comparing to XP, and to Jarred's, Jrobs, and Mihai's rankings)
There's a lot of tweaks to how ELO can be used (K-factor, parallel v.s. sequential, discrete v.s. range...etc.) so I suspect it will take a bit of playing around with the data before we have a sense of wether or not this is a useful stat.
I can't imagine this hitting engineering backlog in the next two or three weeks but I wanted to get in the habit of including you folks as early as possible in potential paradigm-changing issues. If you have the bandwidth would love your feedback/thoughts/ideas...etc.
heyheyjp
tannerwelsh
jeffreywescott
Context
To solve for this, the amount of XP a player gains on a project should be a function of their rank compared to the rank of the players they play with, in addition to their relative contribution and hours.
To do this, we need a separate stat that embodies rating, that's separate from XP.
Having a separate stat that represents rating allows us to:
Proposed Solution
The Elo Rating System is widely used (chess, fifa, online gaming...etc.) as a system for rating two-team competitive games.
Every project retrospective can be seen us multiple two-player 'contests' with each player competing to contribute the most per hour spent on project.
For example, assume a project where the following players contribute the following:
Each player's relative contribution per hour is calculated by dividing their rc by hours:
The retrospective is now broken down into three two-player contests:
As a result of this example, none of the player ratings would change, but all of their XP would increase, with Jeffrey and Tanner's XP increasing by double what Shereef's XP increased by.
Taking another example, where the players have different skill levels and different contributions:
Breaking down the retrospective to three contests:
Shereef wins two contests, and tanner wins one.
Considerations
a. Order games sequentially, and adjust player rankings after each contest. b. Adjust player rankings "in parallel"
Choosing a. gives us a more accurate distribution of rank, but disadvantages players based on how the games were ordered sequentially. Choosing b. is more fair, but does not give us as accurate a distribution.
For example, in the second example above, if Jeffrey loses to Tanner after Tanner loses to Shereef, then Jeffrey's final rating will be lower than if he loses to Tanner before Tanner loses to Shereef.
If we end up using a margin-based ELO (similar to GO) it might make more sense to run the games "in parallel"