search

stacks-archive / app-mining

For App Mining landing page development and App Mining operations.
https://app.co/mining
MIT License
48 stars 16 forks source link

Proposal: Improve impact of no votes on Democry Earth (short term) #33

Closed friedger closed 5 years ago

friedger commented 5 years ago

App Review Democracy Earth is not very transparent about what criteria they use to judge good apps.

Until now, there are still apps that do not receive any votes at all and receive an average score of -1. This impacts the rating for all apps as the likeability is around 0.5 for apps that receive at least 1 vote and 0 downvotes.

Proposal: Give 1 vote extra to all apps and encourage DE to actively downvote "bad" apps.

Impact: Likability is around 0.15 for apps that receive at least 1 vote and 0 downvotes.

https://docs.google.com/spreadsheets/d/1W1-PlyB2d5aqwa_JACGtOp9zLy-zyucFEG1NQO0eWR0/edit?usp=sharing

paula-berman commented 5 years ago

Hi @friedger this is Paula from Democracy Earth here. Thank you for your attention to this point. One clarification here is that we cannot unilaterally "actively downvote bad apps" - voters act on their own will. With that said, the idea behind the current algorithm punishing apps that did not receive any votes at all is to take that as a signal of lack of "popularity". This information is transparently available for you to examine here (see section 2.2 step 2). Currently with the very small amount of downvotes happening in the platform, it doesn't seem like it would be most beneficial to remove this slight punishment of one standard deviation below the average, but happy to continue discussing this point.

friedger commented 5 years ago

@paulamlb Sorry for not being clear. I mean that judges should be encourage to downvote apps that they think should not be ranked high. I understand that DE can't vote.

Indeed, it is the current rule to apply -1 for these apps. My point is that these apps that did not receive any votes impact the scores of the others in an unfair way due to the lack of more votes.

Maybe it is just the way the score is calculated as the -1 in row M,N, and O should be 0.

thedavidlewis commented 5 years ago

A bit off the topic of this thread but related to the point that new apps are at a severe disadvantage. If there is not vote decay, most voters will not vote again so their votes will stay with the top apps not being challenged and new apps being at a disadvantage.

This is not an easy problem to solve as it has many variables:

friedger commented 5 years ago

@paulamlb What is the impact if all apps receive one vote extra? I tried to simulate that and found that not only the apps with 0 votes and with at least one downvote are affected, but also the ordering between the apps with at least one vote.

Is that fair? Is my simulation wrong?

screenshot from 2019-02-22 07-06-21

paula-berman commented 5 years ago

@thedavidlewis thanks for the input! Previous month scores do suffer a decay once taken into account by the app mining algorithm (you can check here: 2.2.2 Future Periods, step 5 Aggregation over time). But you are right that there might be ways to reflect that inside the voting platform as well - note taken :). And indeed as written in the following section of the paper, the matter of emphasizing new apps is crucial - we currently contemplate that by adding a "new" badge to all the new apps

screen shot 2019-02-22 at 12 44 14

and sending an e-mail to voters every time a new batch of apps gets included in the platform, encouraging them to check. Here's a snapshot of the February e-mail:

screen shot 2019-02-22 at 12 40 07

In terms of updates to the platform, our plan is to include categories and one of them will surely be "new apps" so that voters can easily find them!

friedger commented 5 years ago

@paulamlb Could you explain why most apps have 0.5 in likeability? I suggest that all apps get 100 votes to start with. Does that make sense?

hstove commented 5 years ago

@friedger , giving all apps free votes would have a huge impact. Because the vast majority of votes are upvotes, most apps receive "100%" net upvotes. If you gave everyone free votes, the likely outcome is no one will actively downvote those, so the apps with free votes get rewarded the same as those who get real votes. I think that is not good. You should be rewarded for being voted on.

hstove commented 5 years ago

Unless we hear a strong reaction from the community about this, we are going to move to closing this issue in roughly a week.

friedger commented 5 years ago

@hstove I wanted to provide 1 free vote for each app. There shouldn't be much of a difference if you have 0 votes, or 1 or 2. But currently, it is a big difference. If there are 50.000 votes then the 50 free votes shouldn't make a big difference.

I don't understand why it makes such a difference on likeability. @paulamlb Can you explain that?

hstove commented 5 years ago

It makes a big difference because every app with 0 votes goes from 0% likability (and a z-score of -1), to a likability of 100%. Because there are so few downvotes currently, this would effectively negate the likability score, because everyone would have the same score. The likability score is meant to determine how 'liked' your app is, judged by upvotes. It's not fair for apps who have actual upvotes to get the same exact score as an app with zero upvotes.

paula-berman commented 5 years ago

@friedger I echo @hstove here. I fail to see how this creates an unfair situation to the other apps that do receive votes, given that by creating this differentiation we are precisely giving them an advantage: not being ignored has a value. The rationale being that 1 vote is more superior to 0 votes, than 2 votes is to 1 vote. In essence this bump is saying "someone thinks this is a valuable app". It seems more logical and fair to promote the apps that receive that first validation than to artificially promote those who receive none. The former emphasizes a signal that already exists in the platform and comes from the legitimate participants, while I fail to see the logic behind the latter.

Additionally, if that particular logic was deemed unfair, this is a change that would have to be effected directly in the app mining algorithm, not on Democracy Earth's platform. Votes are a signal coming from valid participants - inserting artificial votes changes that definition, thus diminishing the very legitimacy of the votes. Hope this helps clarify the matter :)

friedger commented 5 years ago

Can you explain why the ranks changed between 8,9 and 10 by adding one vote to all of them?

For the change of algorithm, my understanding is that DE provides a score that is at the discretion of the app reviewer. So, DE could change this, not Blockstack.

@hstove, you make an argument against my proposal because there are little downvotes, I make an argument for the proposal because there are so little upvotes.

On Mon, 4 Mar 2019, 20:44 Paula Berman, notifications@github.com wrote:

@friedger https://github.com/friedger I echo @hstove https://github.com/hstove here. I fail to see how this creates an unfair situation to the other apps that do receive votes, given that by creating this differentiation we are precisely giving them an advantage: not being ignored has a value. The rationale being that 1 vote is more superior to 0 votes, than 2 votes is to 1 vote. In essence this bump is saying "someone thinks this is a valuable app". It seems more logical and fair to promote the apps that receive that first validation than to artificially promote those who receive none. The former emphasizes a signal that already exists in the platform and comes from the legitimate participants, while I fail to see the logic behind the latter.

Additionally, if that particular logic was deemed unfair, this is a change that would have to be effected directly in the app mining algorithm, not on Democracy Earth's platform. Votes are a signal coming from valid participants - inserting artificial votes changes that definition, thus diminishing the very legitimacy of the votes. Hope this helps clarify the matter :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/blockstack/app-mining/issues/33#issuecomment-469393929, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYcWQx5kKhDBWZ3eZDvcOxRgBZh8fpkks5vTXe2gaJpZM4bEcgR .

paula-berman commented 5 years ago

@friedger this demonstrates that the algorithm is behaving exactly as expected. The first vote has the additional impact we are discussing, so naturally by adding one single vote, the ranks are impacted.

Regarding the score that we provide, it is a direct reflection of the votes received by each app. We are not inclined to change that for the time being as it is in the best interest of the voting procedure that votes remain being a legitimate signal coming straight from the voters, and not artificially inserted by Democracy Earth. Of course, if new facts and observations lead us to conclude the contrary we are always open to change, but as of now it doesn't look like it would be beneficial by any measure. Our goal is to create something as fair and decentralized as possible - which will ultimately surface the very best applications out there and help them get the support they need. The slight "bump" gained with the first vote on the App Mining algorithm is a reward that is aligned with that goal, thus it wouldn't make sense for us to artificially erase it by arbitrarily meddling with scores.

friedger commented 5 years ago

OK, I'm closing this now.