search

chrismbryant / amazon-confidence-interval

A browser extension which adds Bayesian visualizations to Amazon ratings.
MIT License
31 stars 4 forks source link

Assess the difference between two stats methods #10

Open chrismbryant opened 4 years ago

chrismbryant commented 4 years ago

For each Amazon product, there exists a distribution of star ratings (number of ratings per star value). To compute the probability of positive experience (i.e. satisfaction probability) with the product, we'd like to assign 4 and 5 stars the binary label of "positive" and {1, 2, 3} stars the binary label of "not positive". Given this assignment, we assume that each rating can be viewed as an independent Bernoulli trial with fixed (but unknown) probability of success. Taking this assumption, we can use the Beta distribution to compute a confidence interval on our measurement in a straightforward way.

However, it is not feasible to access the full histogram of star ratings for a full page of products immediately on page load (Amazon locks you out if you fetch these results too quickly, and rate limiting results in a poor user experience). Instead, the only variables we have readily accessible to us are the average star rating and number of ratings for each product. Given these two pieces of information, we can obtain an alternative satisfaction probability and confidence interval by linearly scaling the star rating in range [1, 5] to a success probability in range [0, 1], then building a confidence interval off of that proportion in the same way as before.

What are the consequences of choosing the "average star rating" as a proxy for "proportion of 4 or 5 star ratings"? Can the Beta-distribution-derived confidence interval still be trusted?

aeciorc commented 4 years ago

@chrismbryant @musicin3d , I just noticed this disclaimer under the reviews:

Screen Shot 2020-04-09 at 1 42 11 PM

Could this mean that using the average star rating actually serves us better?

chrismbryant commented 4 years ago

Interesting! I had no idea. I think you might be right then that it’s probably better to use the average star rating rather than the raw star distribution.

musicin3d commented 4 years ago

I have returned from my sulking. I needed some time to accept that all my work was in vain. ;)

Since we've realized the averages serve us better, are we ready to close this issue or are there questions remaining?