Points system should incentive voting for PRs and discourage rejecting PRs

[tnelling]

I mentioned in #33 that the original nomic scoring awarded points based on voting. I'd like to figure out a way to use this to encourage approval and make rejection non-trivial.

If we can come up with something especially clever, perhaps we can keep vetoing around as-is. I'm thinking of a system in which you pay points for rejecting. One problem is the permanence of it. We don't have a good way to store points without triggering another round of voting. I guess the GitHub API does support going through the entire review history, so we could recalculate it every time.

As for approving, I'm thinking you only get rewarded if the PR is merged (and obviously don't get extra credit for approving multiple iterations). We might also want to re-scale the points for creating a merged PR so that being the one to actually create the PR is more valuable than just approving it.

Finally, it would be neat to have a way to reward points to people for contributing to a PR, even if they didn't create it themselves. Obviously this could be done in the PR itself by adding the points in the player directory, but that has the potential to cause tons of merge conflicts.

[jeffkaufman]

Supportive in general

• If you inspect the whole review history you need to be careful not to make too many requests. We get 5k/hr to play with. Possibly we could have some sort of cache where you could run a script which makes a PR that pulls in a new section of history, and then validate could allow this to be merged without approvals after duplicating the work to verify it.

• To avoid merge conflicts in assigning people points we could again use the directory trick: a directory of named bonuses

[jeffkaufman]

To avoid merge conflicts in assigning people points we could again use the directory trick: a directory of named bonuses

implemented in #47 / #33

[pavellishin]

I'm thinking of a system in which you pay points for rejecting.

I think this is a very good idea, but @jeffkaufman makes very good points about our API limits.

Maybe we can do something convoluted like seeing if a user trying to reject a PR has recently (e.g., within the last week) made any commits decreasing their total point value. If not, their rejection carries no weight.

One obvious problem is that PR merge times are pretty slow, so there will likely be times where your rejections won't carry any weight, and you won't be able to do much about it for awhile. Another problem is that people may refuse to approve those PRs for users who are known to reject a lot of PRs to strip them of their veto power. (Which maybe isn't even a problem, but just another layer of meta-gaming.)

[jeffkaufman]

We could extend #49 to allow you to remove your own points without approvals from others. Then you could create a file containing "-1" like players/$name/bonuses/reject-pr-$n or reject-$date to reject a PR in a costly way.

[pavellishin]

Oh, I like that idea.

Although I think I'm with you on the "more PRs, faster" - so I'd rather that be a separate PR. But I also haven't read #49 yet; I'll try to do it now.

[tnelling]

I like the idea of allowing you to decrease your own points without approval for costly rejection. We could combine that with an update to veto power to make rejections only have veto-level power if you pay points.

[tnelling]

I think we're ready to revisit this, if people are interested.

Maybe starting with changes to #49, or creating a new PR that does what @jeffkaufman suggested (allowing you to remove your own points, and only your own points, without approval).

If that was going to function as a sort of super-veto system (or put another way, if we made regular rejections not vetoes and used point-buys as vetoes), we'd need to also add something in validate.py that checks against point-buys with the same PR# to scan for vetoes.