search

stacks-archive / app-mining

For App Mining landing page development and App Mining operations.
https://app.co/mining
MIT License
49 stars 16 forks source link

Define for all reviewers an upper limit on the score #154

Closed friedger closed 4 years ago

friedger commented 4 years ago

What is the problem you are seeing? Please describe. Some app reviewers have an upper limit on the maximum score (before normalizing and applying theta functions), some don't.

How is this problem misaligned with goals of app mining? This encourage developers to focus on improving only on the reviewers without upper limit. For example PH or AW.

What is the explicit recommendation you’re looking to propose? For app reviewers, measuring a value without upper bounds (like upvotes, reach) use a function like MAX / (1 - 1/VALUE)

Additional context See #121

hstove commented 4 years ago

Sorry, I'm not sure that this makes sense. If we defined an upper bound, then it will de-incentivize folks from improving on it. TMUI has an upper bound, but people still have a big incentive to improve there.

friedger commented 4 years ago

@hstove NIL has an upper bound as well. I think the current state makes the score of the reviewers unequally valued. App developers can have a lower score on NIL and TMUI but get into top ranks because of very high scores on AW or PH.

Wouldn't it be much fairer to each reviewer if all reviewers can give a score between 0 and 100?

Indeed, it does not have an immediate impact on the final score as the theta function is applied, but the theta function returns usually much lower values on a bounded range than on an unbounded range because it is much easier to score above average.

hstove commented 4 years ago

The Awario algorithm already does normalization, so it's not between 0-100, but it's generally between -2 and 2, especially after the theta function. Sorry, I just don't see a good way to put a hard cap on this without punishing people who do well in Awario.

I understand your concern around unequal weighting, but at least everyone has to be scored by the same reviewers, so there is less unequal scoring overall. I would prefer to add dimensions to NIL, which would cause a more distributed score from that reviewer.