Players can see their Elo-based rating

bundacia commented 8 years ago

(Depends on #422)

Overview

Players have a stats panel, toggled with a /stats command, that displays their Elo information (and eventually other stats as well).

User Benefits

As a player, I am aware of my relative skill level to other players
As a player, I can investigate a history of my projects and how my skill rating changed over time
Mockups + Flow
1. A player wants to know their Elo rating
2. They issue the /stats elo command
3. The stats panel opens, showing the Elo rating and history
  Elo Rating + History

elo rating v3

UI Notes

Should load the most recent 5 projects for a player. As they scroll down, load more and auto-populate until the player stops scrolling or reaches the end of their project history.

UX Issues

Can players view their own Elo ranking? If so, how?

Yes. By issuing /stats elo command to open Elo detail panel (currently only has Elo rating).

Can players view the Elo ranking of other players? If so, how?

Not a current roster of all players, but they will see the ratings of other players they've been on projects with as part of their Elo history, so that the Elo gain/loss is clear.

tannerwelsh commented 8 years ago

Aright folks, first UI proposal ready for review. @shereefb @prattsj @bundacia can I get your eyes and thoughts on this design?

jaredatron commented 8 years ago

This looks like it would be super useful. Can we ask some Learners what they think?

jrob8577 commented 8 years ago

Would it be possible to make the comments accessible as well, so that we have a history of suggestions we could use to improve our stats?
This will become an awful lot of information to peruse through a panel (even without the written responses from the teams) - is there an alternative UI? Web page?

shereefb commented 8 years ago

@deadlyicon I really don't think we should be pulling in learners' feedback at this stage of the process. Asking them what they think after they deployed it makes more sense to me in terms of the sheer amount of feedback to integrate.

@tannerwelsh I really like the history, showing the hours, contribution and effectiveness. The ELO change on the top right doesn't make sense to me. I think it should be in brackets next to each player:

@deadlyicon - Elo 1298 (+12) @annasauce - Elo 1032 (-6)

jeffreywescott commented 8 years ago

@tannerwelsh I like this.

FWIW, I read the Elo change at the top right as how much the Elo changed for the player who issued the /stats command for that project. If that's correct, I like it. If that's incorrect, then I agree with @shereefb that it doesn't make sense.

tannerwelsh commented 8 years ago

Would it be possible to make the comments accessible as well, so that we have a history of suggestions we could use to improve our stats?

Which comments are you referencing @jrob8577? For this particular issue, my goal is to provide just enough information to make Elo understandable. Actionable text feedback is definitely useful (and will/should be a part of stats UI), but I'm not sure if it is absolutely necessary for this case.

This will become an awful lot of information to peruse through a panel

True, and we're considering other UI options for these reasons and more. It would require a lot of vertical scrolling to see the entire history, but it is at least one-dimensional. You only need to see the neighbors of any given element to understand the changes in Elo rating between projects.

tannerwelsh commented 8 years ago

The ELO change on the top right doesn't make sense to me.

As @jeffreywescott guessed, it is intended to represent the change in Elo rating for the issuer of the /stats command (the subject). Because it caused confusion, I'll revisit it.

tannerwelsh commented 8 years ago

Thanks for the feedback folks. I'm going to make a few changes and will notify you when the second draft is ready.

bundacia commented 8 years ago

Since ELO going up or down is not correlated directly to my own progress as a player, I think we should make it a little less prominent. As a player, I shouldn't care if my ELO dropped 9 points. I care if I went up or down in the rankings. Maybe the ranking or percentile should be the more prominent thing we're showing here, and we can show how that went up or down with each project. ELO seems almost like an implementation detail. I'm not sure if we need to show that to players at all, but if we do show it, we shouldn't make it look like the most important number. I think that will cause confusion.

Maybe we should call this "ELO Ranking" and show percentile everywhere we're currently showing raw ELO and "Percentile Change" instead of ELO change.

heyheyjp commented 8 years ago

@tannerwelsh: this feels significantly inconsistent with the design of other flex-panel UIs. The black background is perhaps the most jarring - can this be revisited or would you mind explaining the rationale for breaking with the design language (default react-toolbox styling, for better or worse) that's already established?

tannerwelsh commented 8 years ago

As a player, I shouldn't care if my ELO dropped 9 points. I care if I went up or down in the rankings.

This is a good point @bundacia, although I have a counter-scenario in mind that I think makes Elo rating as important as position. This is that when new learners join, they will (in the near future, at least) be joining en masse, so an existing player's rank (position) will change dramatically and quickly simply by the fact of new learners joining.

Because the ranking number depends on factors outside of a learner's experience (i.e. the total number of learners in the chapter), it will change in ways that are unexpected. Percentile will also change in unexpected ways for the same reason, but less dramatically. Elo is the only number that changes solely as the result of a learner's experience in projects.

A second reason is that we'd like to (eventually) get to a place where Elo numbers really do mean something about the skill level of the learners, in the same way that a Level 8 Mage means something real to a fantasy game player (pardon the abstract analogy), even though the number 8 doesn't signify anything outside of that context.

All that being said, I'd like the project elements in this UI to reflect as much important change as possible in order for it to be useful feedback. So perhaps each project card should show change in all three at equal levels of visual hierarchy: Elo change, rank change, and percentile.

tannerwelsh commented 8 years ago

this feels significantly inconsistent with the design of other flex-panel UIs.

thanks @prattsj - I'll adjust to use standard react-toolbox components in the second draft. Mostly wanted to keep the first draft simple but with high contrast to emphasize visual hierarchy and element identity.

tannerwelsh commented 8 years ago

Second draft is ready. Changes:

Simpler (less information presented overall, fewer duplicates)
Uses more standard react-toolbox (Material UI) components
Single element type in the list (just projects, with every player included)

elo rating

Thoughts @prattsj @bundacia @shereefb?

shereefb commented 8 years ago

I love it. What happens when there's a 20 project history. Pagination? Infinite scroll? Long scroll?

Professional players already have over 40 projects probably. On Wed, Sep 7, 2016 at 11:50 AM Tanner Welsh notifications@github.com wrote:

Second draft is ready. Changes:

Simpler (less information presented overall, fewer duplicates)

Uses more standard react-toolbox (Material UI) components

Single element type in the list (just projects, with every player included)

[image: elo rating] https://cloud.githubusercontent.com/assets/709100/18324328/ebe27c9e-7509-11e6-98a4-0d876661a160.png

Thoughts @prattsj https://github.com/prattsj @bundacia https://github.com/bundacia @shereefb https://github.com/shereefb?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LearnersGuild/game/issues/425#issuecomment-245379460, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGoAWQLhgySjREHKerFkHvb5dIrFr1cks5qnwdZgaJpZM4Jnfv2 .

"When they tell you to grow up, they mean stop growing.” ~Tom Robbins

heyheyjp commented 8 years ago

@tannerwelsh: would love to further discuss the idea of position/ranking. feeling a good amount of tension around it. seems like it would be nice if we could promote individual achievement without setting things up as a "race to the top"...

heyheyjp commented 8 years ago

Other thoughts:

love how much cleaner and consistent this feels :)
wondering if we shouldn't include cycle number next to project names
there's something that feels like it's missing somehow w.r.t. the connection between projects and changes in Elo ratings...i think i missed how the issue of run-in-parallel vs. run-in-series was resolved for Elo changes at the end of a project cycle. i think there was some consensus that the run-in-parallel approach was fairer...but i'm getting the impression that we've chosen the run-in-series approach instead (still only just digging into Elo rating implementation, so maybe i'll find out soon)
do we need to show the date at the top? can it not be assumed to be the current date, valid as of the last change/change at the top of the list?

heyheyjp commented 8 years ago

Moar thoughts:

positions N (of M)...feels like non-MVP and is expensive enough to implement that i think we should consider punting on this (and perhaps rethinking it entirely - see above comment)
same for percentile re: MVP. expensive enough to warrant consideration of punting given how close we are to the 19th and how much more there is to do

heyheyjp commented 8 years ago

Even moar thoughts:

i think it's likely that more than one person will feel a little confused about the 0.57 effec. blurbs. i get it because i understand the basics about how Elo works. learners probably won't grasp the point - at least not at first glance. can we make this more plain/clear?
what do you think about being low-key about linking this to the Elo name? ratings are shifting so frequently...maybe the less learners get attached to a specific ratings system, the better? transparency is essential...not recommending that we obscure the fact that these ratings are based on Elo...just maybe that we don't have to publish them as the "Elo" rating and instead could message that this is their LOS rating...which is currently based on the Elo rating system.

bundacia commented 8 years ago

Still feels problematic to me that the raw ELO rating is the most prominent statistic in this view. We're only using ELO to rank players and compare them to each other, and I think that that is most clearly communicated by an actual rank and/or percentile. Presenting raw ELO like this makes it look like something it's not.

bundacia commented 8 years ago

Visually this is a big improvement 👏

tannerwelsh commented 8 years ago

Updated the description. This looks like a good MVP to me. Unblocking now, and will respond the feedback...

tannerwelsh commented 8 years ago

re: @shereefb

What happens when there's a 20 project history. Pagination? Infinite scroll? Long scroll?

Infinite scroll. Added to UI notes in description.

tannerwelsh commented 8 years ago

re: @prattsj

further discuss the idea of position/ranking

Would love to as well. I removed this feature for now, since I agree that it is not MVP. Punting for a later discussion.

seems like it would be nice if we could promote individual achievement without setting things up as a "race to the top"...

Agreed. And, objective skill measurements are hard. Maybe a separate issue, because I think it's unfortunately not a solved problem. Seems like the industry in general struggles with this.

include cycle number next to project names

Good idea, not MVP IMHO. For now a date should be enough to contextualize the data.

w.r.t. the connection between projects and changes in Elo ratings...i think i missed how the issue of run-in-parallel vs. run-in-series was resolved for Elo changes at the end of a project cycle.

AFAIK, we haven't yet found a way to actually do run-in-parallel Elo rating. If that's possible, then great.

This UI works for both: all it shows is the changes to Elo for all players at the end of the project (i.e. the result of all "matches"), and thus doesn't care whether the "matches" were run in series or parallel.

do we need to show the date at the top?

Nope. :) Removed.

positions N (of M)...feels like non-MVP [and] percentile

I agree. Removed.

i think it's likely that more than one person will feel a little confused about the 0.57 effec. blurbs.

What do you think of the change?

0.65 effectiveness (25% contrib. / 38 hrs.)

instead of

25% contrib. | 38 hrs. | 0.65 effec.

what do you think about being low-key about linking this to the Elo name?

This relates to a larger issue/tension of mine, which is that our stats don't connect well into a consistent story. They are all interesting data points, with varying degrees of actionability, but they don't have much organization, or layers of abstraction that could help learners think about them more effectively, and move between layers of aggregation.

Anyway, for now game mechanics is operating on a "make all individual stats as clear/transparent on their own first, then aggregate/organize/abstract later" strategy. Elo (or something similar) will likely become a part of a family and/or aggregation of "Skill" or "Mastery" stats, but for now I'd like to keep the fairings off, so to speak. Let the machinery speak for itself as much as we can without creating an unreasonable amount of clutter.

In case you didn't see, Elo rating has been added to and explained in the playbook: http://playbook.learnersguild.org/Game_Manual/Stats.html. It's not perfect, but at least learners can directly explore and engage with the Elo system and learn about it/critique it/suggest changes with us.

heyheyjp commented 8 years ago

u da best, @tannerwelsh. all looks/sounds good. ty.

tannerwelsh commented 8 years ago

Still feels problematic to me that the raw ELO rating is the most prominent statistic in this view. We're only using ELO to rank players and compare them to each other, and I think that that is most clearly communicated by an actual rank and/or percentile.

Sorry @bundacia, I disagree. We're using Elo to group players (for pool/project formation), which is not the same thing as a ranking. Also, I don't know exactly how we'll be using Elo in the future, so I've stripped the ranking/percentile altogether for now. (see comments above).

tannerwelsh commented 8 years ago

thanks @prattsj! and thanks to all ( @shereefb @bundacia @jrob8577 @deadlyicon @jeffreywescott ) for the inputs. love to see something improve so noticeably so quickly.

heyheyjp commented 8 years ago

@tannerwelsh @shereefb: one more thing..

AFAIK, we haven't yet found a way to actually do run-in-parallel Elo rating. If that's possible, then great.

Could we just have the result of the match between each team member be based on their Elo rating at the start of the project? Example...if player A has an Elo of 100 at the time that they begin 5 matches (at the start of the project(s) on which they're working with 5 players)...when it comes time to compute match results, we run them in parallel by using that starting Elo of 100 for all 5 matches.

Am I missing something?

shereefb commented 8 years ago

That wouldn't work. After running 5 games in parallel you would have five different elo results for each player. What would you do with them then? On Wed, Sep 7, 2016 at 9:10 PM SJ Pratt notifications@github.com wrote:

@tannerwelsh https://github.com/tannerwelsh @shereefb https://github.com/shereefb: one more thing..

AFAIK, we haven't yet found a way to actually do run-in-parallel Elo rating. If that's possible, then great.

Could we just have the result of the match between each team member be based on their Elo rating at the start of the project? Example...if player A has an Elo of 100 at the time that they begin 5 matches (at the start of the project(s) on which they're working with 5 players)...when it comes time to compute match results, we run them in parallel by using that starting Elo for all 5 matches.

Am I missing something?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LearnersGuild/game/issues/425#issuecomment-245488547, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGoAUs5RZI4rbrGvB8RiG14q90nK_J6ks5qn4qqgaJpZM4Jnfv2 .

"When they tell you to grow up, they mean stop growing.” ~Tom Robbins

heyheyjp commented 8 years ago

Is there some reason we couldn't/shouldn't apply them all?

heyheyjp commented 8 years ago

There would be 5 different adjustments made to each player's Elo ratings either way. The difference is that these additions/subtractions would be based on a fairer/truer representation of the matchup (starting Elo vs. starting Elo) rather than dependent upon the order in which match results are calculated (basically arbitrary in the prototype algorithm, AFAICT), which also makes it possible/easier to reason about results.

bundacia commented 8 years ago

We're using Elo to group players (for pool/project formation), which is not the same thing as a ranking.

But ELO is a ranking. We're using that ranking to group players into pools, but that doesn't change what ELO means. We may not know what uses we'll have for ELO in the future, but how we use it won't change the fact that it represents your ranking among the rest of the players in the chapter.

It seems like I'm alone in this tension and this is clearly out of my domain. For some reason I just seem to have a really strong opinion on this one. Anyway, since the blocked label has been removed I'll take that as a sign that it's time to move forward.

heyheyjp commented 8 years ago

Let's be careful not to conflate rating with ranking. There's is an important distinction there. Elo is a rating system, not a ranking system.
Since @tannerwelsh has agreed that the percentile and position figures can be left out for v1, it seems like potential confusion around the relationship between elo rating and rank or percentile isn't an issue anymore, no? At least, not for this iteration..

tannerwelsh commented 8 years ago

@prattsj let's move the parallel-vs-series discussion to the implementation issue #422, since it doesn't directly impact the outcome of this issue.

Let's be careful not to conflate rating with ranking.

👍 to this. Important distinction.

@bundacia I hear your tensions, and please bring them to game mechanics. I removed the blocked label as a way to indicate "this UI spec is ready for implementation".

tannerwelsh commented 7 years ago

Needs UI redesign to be consistent w/ #615

jeffreywescott commented 7 years ago

Marking blocked pending updated UI design.

jeffreywescott commented 7 years ago

We will revisit this after this is done: https://app.clubhouse.io/learnersguild/story/106/stats-ui-v2

LearnersGuild / echo