Status issue curation and communication by bounties

[oskarth]

Preamble

Idea: PROC010
Title: Status issue curation and communication by bounties
Status: Draft
Created: 2017-10-11

Summary

As part of scaling Status, we use Status Open Bounty. To use it effectively we need to curate issues, put bounties on them, and ensure proper communication with newcomers.

Vision

We are already doing this work, and since we want to continue doing it we should formalize it into a swarm. This will probably require high commitment from some people (QA, Community) and low commitment from a larger number of core developers.

We want to ensure we always have a bunch of issues available for people who want to contribute. These should be more than just translations, and vary in bounty size.

Specifically, Status Open Bounty as a standalone project is out of scope for this. This is about us using Open Bounty for status-react, status-go et al. issues, although there's likely to be some overlap.

This is a process, similar to how we do code reviews and QA now, that we want to install in order to scale Status. It requires:

(a) Curation of issues. Not all issues make for great bounties. It is particularly important issues are well-defined in terms of acceptance criteria, and ideally shouldn't require too much context, and no special permissions. They should also ideally not be time-critical, as these issues are best done by core contributors working full-time in dedicated swarms.

(b) Ongoing communication with developers. We want to make sure they get a warm welcome, and that we help them along the way. Additionally, if a bounty issue has been open for a long time we should figure out why and how we can change this. We also want to increase repeat contributors.

Both of these activities and the swarm as a whole has an eye towards improving our two core metrics:

• monthly active contributors
• daily transacting users (of SNT)

Swarm Participants

• Lead Contributor: @andytudhope
• Testing & Evaluation: @annadanchenko
• Contributor: @pablanopete (10-15hr/week)
• Contributor: @oskarth (~5hr/week, similar to code review)
• Contributor: @tiabc (~5hr/week, similar to code review)

Requirements

3 core contributors, 2 from community and one from QA. 1-2 devs from both status-react and status-go to help create, edit and otherwise curate issues just 5hrs/week.

Goals & Implementation Plan

1. Channel created with initial members.
2. Create guidelines for good issues to bounty for devs, QA, and community.
3. Curate ~10-20 issues in each repo initially (by 22/11/2018).
4. Iterate on feedback process to improve flow and document in wiki (along with fixing the current QA docs there, which Anna is doing).
5. Figure out useful ways to pull data from the db into current spreadsheet in order to automate as much as possible, without losing personal touch that community requires.

Minimum Viable Product

Goal Date: 2017-11-19 Description: A swarm with Slack channel and instructions/flows set up for proposal of bounty issues, creation of bounties, testing etc. Labelling system on issues so that there is an easy flow from developers -> QA -> community and back.

Iteration 1

Goal Date: 20/11/2017 Description: Get access to DB so that we can pull the data we need on contributions. Set up automated processes for creating, managing, accepting and confirming bounties. Create Open Bounty Twitter Bot.

Iteration 2

Goal Date: 06/12/2018 Description: Discuss with @tpatja the issue.updated event and whether we can access that from the DB - still no answer there. We have already started a GH project to track issues using a kanban board to limit the co-ordination and communication costs internally. We need to make sure everyone is using this and that it is working as expected. Keep on gathering data and providing to Jonny for analytics and blog posts etc. Co-ordinate wth Arash to get more Riot bounties in, as well as onboard other organisations. This will also mean helping with basic filtering of issues on SOB (https://github.com/status-im/ideas/issues/48) and co-ordinating with @3esmit and @tpatja to see when SOB can implement an admin dashboard, allow multiple contributors to a PR, allow multiple admins to manage and confirm payouts, and set time limits on bounties.

Iteration 3

Goal Date (start): 05/12/2018 Goal Date (end): 01/03/2018 Description: Track actual metrics to improve our understanding of how well this swarm is achieving its goals.

1. Automation of bounties!! We need to make it so that people can simply assign a size label to their bounty and have it automatically funded from a (super) hot account that they only maintain a small balance in and top up every so often, rather than manually funding each and every bounty. This has been a request from a number of projects, most notably Aragon, who have actually already written a bot for us: https://github.com/aragon/autobounty

2. 10% week over week growth in available issues for all Status repos, as well as in status-go and status-react independently. Currently 11 bounty issues in status-react and 5 in status-go. Do this for 3 months and we are at 30x, then re-evaluate since we might hit on financial limits.

3. 10% week over week growth in new contributors for all Status repos, as well as in status-go and status-react independently. After 1 month of real tracking introduce 3rd metric which measures and grows recurring contributors.

Get this data for last few weeks. Plot it over a timeline. Plaster it all over #general on a weekly basis. This achieves three things:

(a) communicates success (b) clarifies deficiencies, e.g. this repo failed to meet growth target this week - why and how do we fix? (c) forces automation naturally, e.g. instead of "automate" or "bounty all issues" at once we can gradually introduce processes that actually solve bottlenecks.

4. 5 other recognisable projects using SOB by 28/02/2018 (Aragon and Riot count).

Supporting Role Communication

Post-Mortem

Copyright

Copyright and related rights waived via CC0.

[andytudhope]

Hi @oskarth! I am happy to lead this if everyone else is OK with that. I think Anna might want to focus on QA, and only devote the time Testing & Evaluation requires here. That said, if I am wrong, it's worth noting that Anna is more experienced than me and currently has deeper insight into the issues in status-react and status-go, so would make a better lead if she does want to take point. Would be an awesome way for me to learn more about the code-base and sharpen my own clojure and go skills!

[annadanchenko]

(a) Curation of issues. I can do it + communication and coordination with a core team, so we understand the SOB mechanism and have more and more bounty issues with minimal administration effort on top of this. Guess 10 hrs per week might be enough (b) - assumed it's external developers who contribute - suggest @andytudhope can do this in close cooperation with SOB team and me

[oskarth]

@andytudhope @annadanchenko Cool, go for it! Do you want to setup channel etc? As a dev would like to have clarity in what specifically I can do to get other people to do specific pieces of work I know need to be done but they don't have to be done by me.

I think it'd also be good if we have 1-2 devs from both Clojure and Go team who can help @annadanchenko select issues, especially since a bunch of these might be technical and not necessarily user-focused.

[pablanopete]

Very happy to help with this initiative as well @oskarth @annadanchenko @andytudhope

[andytudhope]

OK, so Anna, Hutch and I will collaborate on this. @oskarth and @tiabc will help just a few hours a week to curate the best issues for bounty following O's outline above (particular w.r.t time-critical issues that should rather be solved by core contributors).

Anna has devoted 10 hours/week to curating correct issues with the team. I will devote 10 hours/week to ensuring communication with all devs, capturing their details and following up in GH and Riot. Hutch will support these efforts, though has not yet detailed a time commitment. Suggest one call per week to identify bounty issues that are not being attempted and try to figure out why, to discuss issues under review, and generally keep on top of things.

Most of the initial processes are set up, but far too manual. @pablanopete it would be really great to see if we can automate the contributor spreadsheet. @oskarth is there not just a way to pull this stuff (GH username, email if available, website etc.) directly from the SOB backend? We already have the analytics from there to assess accurately the two stated goals:

monthly active contributors daily transacting users (of SNT)

and would be nice to see if we can enrich that, or just make it more accessible to the team.

[oskarth]

is there not just a way to pull this stuff (GH username, email if available, website etc.) directly from the SOB backend?

@andytudhope I believe we have a bunch of this data available, yes. If you give me a list of the data you think would be useful I'll see what I can dig up later this week. Some (website) might require some additional work but I suspect basics (username, issue, payout) can fairly easily be dumped from DB. We can then create a cron job to send report of this on a weekly basis, say.

[pablanopete]

I can devote at least ~10 HR PW as well to communication with devs / followup and spreadsheet curation. ~5 HR PW to issue curation after discussion with @annadanchenko @andytudhope and definition of issue criteria

[annadanchenko]

@oskarth @tiabc @andytudhope @pablanopete My vision on the bounty process in the core team. So, any core team member can submit a proposal. Text as it's explained to the core team member (developer, tester):

1. If we have something to delegate to the contributors - we can do this
2. Issue for the bounty should satisfy these conditions:
   • we are ok to wait undefined time for the PR
   • it's described in a good enough way, so external contributor can understand. If it's a bug then there are clear steps to reproduce, actual and expected results. In case of status-react project - there is a link to TestFairy session with logs/video
   • we have clear acceptance criteria: how do we check that it's done?
   • we can estimate its complexity in XS, S, M, L or XL style (XS - tiny issue to fix, XL - lots of research and work to be done). Most of the time we will use S, M and L labels.
3. If you have such issue (or will create one) then tag it with a
   • label "Ready for bounty check"
   • label of complexity
4. At least 2 more team members agree we need it. Comment or +1 in the issue from extra 2 team members are OK. ? Need leads to approve too?
5. As soon as it's done you can just wait or ping in the Slack channel 10-ob-bounties with issue number. As a result, Anna or someone else from the "Bounty curation team" (see https://github.com/status-im/ideas/issues/10 ) will add a label and send SNT tokens to it.
6. Please, do not add label "bounty" on your own - it will result in 0 bounty shown to the contributors, so "Bounty curation team" will make sure that issue is listed and gets SNT almost in the same time.

[tiabc]

1. Let's specify concrete names for bounty sizes, say: bounty-size-XL? Are there any borders between sizes? If we don't have them, the estimations will be pretty subjective and approximate.
2. Can we change the issue size on the go in case it had been expected to take little time but turned out to be really complex?
3. Do we have any written instructions for assigning bounties except for this comment? If not, let's create it. If yes, let's add missing parts there.
4. I'm in the bounty curation team. What steps shall I follow to send SNTs to the issue?
5. Let's rename Ready for bounty check to something like bounty-awaiting-approval or awaiting-bounty-approval or awaiting-bounty-check or anything else consistent with other labels format?

[tiabc]

6. I don't think we need any lead approval. We're striving to a scalable and consensus-based team so the lead's position becomes sort of deprecated, as I can see. Also, we don't want leads to be a bottleneck and distract to stuff like that.

[oskarth]

Clarification on distinction between this swarm and SOB:

1. SOB is a separate division/team with goals to have devs and organizations in general. This would still exist even if Status wasn't a consumer of SOB.

2. This swarm is about Status as a consumer of SOB. Even if SOB didn't exist Status would still benefit from a bounty program like Gitcoin.

[naghdy]

Have there been progress to this? We seem to be running out of PR's for people externally to be involved in, but I doubt it's because we don't have work to do.

[andytudhope]

@naghdy so this is definitely something we need to work on as it is the one part of our process that we can't directly control - we require the devs in either status-react or status-go to move issues into the appropriate column in their ZenHub boards (Bounty-Check in status-react, Goteam stil needs to make one) so that we can review them, edit the issue for clarity and prepare it for bounty. We require some input from the devs themselves though - I have been trying to encourage both teams to submit smaller/time consuming but not hard issues, or break down current ones into simpler tasks, but have not had much luck with that.

[oskarth]

@andytudhope As I said a week or two ago, we need clear metrics and graphics that illustrate this on a regular basis, otherwise it is too vague.

The iteration goals are still tasks, not actually goals in terms of % growth of bounty issues / contributors. To me this is the main task of the swarm leader. I'd be happy to assist.

[oskarth]

Concrete suggestions for metrics

1. 10% week over week over growth in available issues for all Status repos, as well as in status-go and status-react independently. Assume 10 issues first week. Do this for 3 months and we are at 30x, then re-evaluate since we might hit on financial limits.

2. 10% week over week growth in new contributors for all Status repos, as well as in status-go and status-react independently. After 1 month of real tracking introduce 3rd metric which measures and grows recurring contributors.

Get this data for last few weeks. Plot it over a timeline. Plaster it all over #general on a weekly basis. This achieves three things:

(a) communicate success (b) clarifies deficiencies, e.g. this repo failed to meet growth target this week - why and how do we fix? (c) forces automation naturally, e.g. instead of "automate" or "bounty all issues" at once we can gradually introduce processes that actually solve bottlenecks.

To me this is the proper way of setting targets and it shouldn't take too much time, it just requires someone to do it regularly.

They also tie directly into the metrics mentioned in org paper.

@andytudhope @annadanchenko

[andytudhope]

Perfect, will make that a reality, thank you for the suggestions @oskarth - inspiring as always 😉

We have 11 bounty issues in status-react and 5 in status-go, so don't really need that assumption - can just start with the real numbers. Otherwise, the rest sounds good. Particularly like the part of tracking recurring contributors (though it seems like most of these so far are falling down further into core contributor positions).

[naghdy]

Those metrics seem solid, this is great to see.

Thanks @andytudhope and @oskarth for driving this.

[andytudhope]

This swarm has now been closed and absorbed into OpenBounties itself, specifically https://github.com/status-im/ideas/issues/59.

You can read the full report here: https://github.com/orgs/status-im/teams/status/discussions/9