Open stackatron opened 4 years ago
Mysterious reviewer sounds like a good idea in theory but here is the catch.
I agree with Josh's point on Mystery Shoppers being users. That should be verified. We have had a problem with TMUI providing testers who claim that they are a specific category of users but are not.
no conflict of interest
This is explained in the proposal. I know it is not perfect, but there is some attempt to review the people.
Solves the same problem as https://github.com/blockstack/app-mining/issues/174 with a different solution.
Overall problem:
What we need is a way to reward meaningful progress on apps. This is hard to objectively measure. And if we did objectively measure it, it would likely be game-able.
Game-able objective measures:
Here are the suggestions from the proof-of-progress thread simply to illustrate the point:
In my opinion our true objective is to reward apps that provide value to users as measured by retention. Since we are delayed on retention, I'll propose a temporary, secondary objective: Reward apps that make quality improvements that benefit users. This is a subjective goal, and so I suggest we use subjective measures for scoring.
Mystery User Reviewer
@jcnelson thought of this reviewer. He was inspired by Mystery Shoppers that provide quality metrics for retail experiences.
Bootup
Monthly run
Mystery User Reviewer uses random selection to pick 5 Testers. Each Tester will:
Scoring
Tallying likes (+1) and dislikes (-1) across all apps to create raw score.
Abstract considerations:
On the process side of things: