search

protontypes / LibreSelery

Continuous distribution of funding to your project contributors and dependencies. Integrated into GitHub Actions
GNU Affero General Public License v3.0
124 stars 18 forks source link

Weight idea: Consider the fraction of total commits per contributer #132

Open fdietze opened 4 years ago

fdietze commented 4 years ago

Another weight idea:

(number of commits I made) / (total number of commits)
krux02 commented 4 years ago

This was one of the first ideas for weights, but it was rejected. People who want to increase their salary will massively increase their commit count without actually improving much. For me this weight feels as good as measuring the amount of weight somebody contributed to an airplaine for their salery. Even though if an airplane is finished, this metric would not be too bad, but as an up front work encouragement it would have pretty negative effects.

fdietze commented 4 years ago

Please elaborate more, why it's bad.

Ly0n commented 4 years ago

Let's just look at our project. When you click on contributors on Github you will get this ordering list: https://github.com/protontypes/openselery/graphs/contributors

It looks like @kikass13 has only contributed a little but acutely he defined the software architecture in many ways. When we order the list by additions it looks better but still @kikass13 is underestimated. @mw-mw-mw has a lot of additions since she added the logo. Another quantitative approach that maybe could be better. How many successful merges / pull requests are created with commits of these contributors.

Ly0n commented 4 years ago

The problem with my proposal is that I pushed many times directly to the master. You acutely have to look into the PR and keep out the direct pushes to the master.

fdietze commented 4 years ago

I thought that the general idea was, that many different metrics are combined, because every metric in itself is not sufficient.

kikass13 commented 4 years ago

regardless of my personal involvement in anything. We can hopefully agree on some wrong interpretations of hard metrics:

I like the idea of soft metrics like (mentioned by @Ly0n):

kikass13 commented 4 years ago

I am also not a fan of overall amounts of something. It is not feasible to compare 1000 apples against 1 pear. I would rather prefer a boolean approach degrading with time.

consider a list of users [A,B,C,D] and a list of metrics [M1, M2, M3, M4]:

fulfilled_metrics = {}
for user in user:
   for metric in metrics:
       bools, timestamps= DOES_FULLFILL_METRIC_REQUIREMENTS(user, metric)
       fulfilled_metrics [user, metric] = (bools, timestamps)

and then apply an linear or exponential filter to degrade the fulfillment of every metric with time passed. For example:

krux02 commented 4 years ago

Just as an idea for a metric, a metric could take the activity change (first order derivative) into account to encourage people to do more than last month. I am not suggesting though that this is a good metric, The point is, the metric are always complex and you always have to think about people who try to exploit them to appear very active when in reality those people contribute as little as possible, or even harm the project in practice.

kikass13 commented 4 years ago

@krux02 I agree

seems like a nice thing to add .

The point is, the metric are always complex and you always have to think about people who try to exploit them to appear very active when in reality those people contribute as little as possible, or even harm the project in practice.

That is exactly why these metrics have to depend on things (commits, actions, reviews) [what @Ly0n said] that have to do with merges and should not involve direct amounts of things (1000 commits vs 10). Instead, these should be a checklist(true,false) of weights that can be achieved (cumulative)

In my opinion, we should start looking at those metrics as "goals to achieve". A developer either achieves something or he doesn't. If so, the weight of this achievement will be added to the sum of all other weights, therefore it is more likely for a person with more achievements to receive a payout.

To ensure that these achievements are not all flagged as true after a period of time (because at some point, someone will eventually tick all the boxes), these should "degrade" over time. That way, an achievement which would correspond to a 1.0 to the sum of weights for a specific user would be downgraded if that achievement was long in the past and throttle at a minimum of (for example) 0.01 [1%] so that everyone still gets a chance, even if his contributions are way in the past in relation to other fresher contributions of other contributors (users).

Ly0n commented 4 years ago

you always have to think about people who try to exploit them to appear very active when in reality those people contribute as little as possible or even harm the project in practice. Yes that's true, but you should keep multiple boundary conditions in mind:

  1. People have a public account on Github and they have a high interest to keep the history nice and shiny.
  2. We just consider things that happen on the master branch and normally you do have multiple people approving and reviewing the code. Especially on the larger projects and OpenSelery is interesting for larger projects with multiple people donating and contributing.
  3. We publish the donation history with every push to the master in the wiki. When people misuse the system it will be visible for everyone.

And one important sentence in this context I also want to cite here from our README: The distribution will never be fair since fairness can not be objectively measured. I personally have a lot of commits on the README and to test the CI integration on master branch.
This was necessary since I needed to test the CI integration and that only works live. Many people might think that this was not necessary and created many commits on the master branch we don't need.

To get next to something that many people will call "fair" you need to experiment with different weight calculation concepts and constant feedback of your contributors. We have here the change to experiment with that so that others can learn from our experiences. That's why we should discuss about every weight and even integrate it despite it may not be useful for OpenSelery. We can just show that by giving it a low weight. Imagine a travel blog financed with OpenSelery and hosted on Github by a group of people. Here the lines of "writing" could more important than the pull requests.

In my opinion, we should start looking at those metrics as "goals to achieve": Yes, that's a good point. The Github API should give us here everything we need. Also, the people that are contributing to the "Issue" discussion should be rewarded somehow. Reporting and describing an issue in the right way can be as important and complex as solving it. When multiple people are involved in reaching a "goal" together like @kikass13 suggested we should also reward them as one team.

kikass13 commented 4 years ago

@Ly0n

People have a public account on Github and they have a high interest to keep the history nice and shiny.

  • I do not agree

We just consider things that happen on the master branch and normally you do have multiple people approving and reviewing the code. Especially on the larger projects and OpenSelery is interesting for larger projects with multiple people donating and contributing.

  • I do agree

We publish the donation history with every push to the master in the wiki. When people misuse the system it will be visible for everyone.

  • the tool should not have to be "overwatched" to be useful. If you cannot trust your tool, you will avoid it.
kikass13 commented 4 years ago

@Ly0n Objective morality can be achieved if the objective is clear. If we boil down the metrics in the way i described earlier, therefore by changing the "things that improve you're weight in relation to others" [which gives you a higher chance of receiving money] to simple tasks, we inherently create a situation where:

We do not have to worry about fairness, because we simply let users tick their respective boxes. They cannot really abuse that because once they "did a thing" (contribution type = true) they cannot "benefit" from doing it again. The user will be punished if he does not do it again in a timely manner (because his contribution will degrade over time if the project is moving on).

We therefore minimize the "damage by exploitation" to a simple boolean flag with simple validity checks.

kikass13 commented 4 years ago

I have found a nice little random blog [CLICK] which sums up to the following:

Ly0n commented 4 years ago

The artical you linked has also many other good ideas:

Employees are encouraged to vote for staff members who have done a great job and rewards are distributed accordingly.

You could use emojis to upvote and downvote issues and use that as an additional bonus on the weights when solving the issue.

if employees who do the same job and get the same results get different rewards, it’s abundantly clear what’s going on. An automated platform with prearranged rewards for certain achievements avoids this.

Where is such a platform? Does somebody know about a company that is doing that besides our project in a real automated way?

@kikass13 the combination of multiple soft contributions and a degregration time could be a good next weight to work on. We have to check what the Github API gives us here and how many request we will need for that.