Hidden Gems negative vote bias is too high

250 / Steam-250

:video_game: Generates the Steam 250 website.

https://steam250.com

GNU Affero General Public License v3.0

54 stars 7 forks source link

Hidden Gems negative vote bias is too high #9

Closed Bilge closed 6 years ago

Bilge commented 6 years ago

Consider two games, both with 100% approval rating. The first has 235 votes, the second has 181 votes. The first has 28K players, the second has 6K players. How should they be ranked against each other?

screenshot-2017-12-17 hidden gems of steam

It may surprise you to learn that not only is the second game (Blue Revolver) ranked above the first (Mindshow), but they're three places apart! So why is that? To find out we have to pluck a few more numbers. Knowing that 100% is a rounded figure, we can start by looking into the raw scores.

	Blue Revolver	Mindshow
Positive	181	234
Negative	0	1

That singular, negative review carries so much weight that it drags the game down three ranks, despite having more reviews overall and almost 5x more players.

@woctezuma I think we need to adjust the weighting to address this.

📓 Note: this data was generated from dump #134.

woctezuma commented 6 years ago

I think it is the intended behavior, because the ranking for hidden gems tries to favor games:

of high quality (which is the case for both examples, as shown by the reviews)
yet a surprisingly small playerbase (in this regard, Blue Revolver is more hidden with 4-5 times fewer players, so it should be ranked higher).

Bilge commented 6 years ago

Are you saying number of players actually sinks games rather than raises them?

woctezuma commented 6 years ago

Yes, there is a trade-off between:

Wilson score (review score taking into account the number of reviews),
and number of players (as estimated by SteamSpy).

That being said, to avoid giving too much influence to that one negative review for Mindshow, one could decrease the weight for the Wilson score (currently $wilson_weight = 1.96), which is akin to decreasing the confidence value (currently, confidence is 95%, see table on Wikipedia).

Bilge commented 6 years ago

It seems either increasing or decreasing the confidence interval just moves them further apart. Decreasing it actually places Blue Revolver at #1. Anyway, it seems you're still happy with the list after bringing this to your attention, so there's no need to make a change. As far as I'm concerned this is still your list so if you're happy, I'm happy; I just wanted to bring it to your attention. 👍