mandatory summaries for long RFC threads

[nikomatsakis]

Over in another issue, we discussed the idea of altering how RFC thread discussions work. Roughly speaking, the idea is to get more in the habit of having a more formal notion of collecting concerns and responding to them.

The current thinking is that when an RFC thread gets long enough, it would be locked, a summary prepared, and a new RFC created that begins with the summary. The summary should convey the current thinking of the team or decision makers, ideally. The goal is then that people responding to the summary should aim to make new points or point out mistakes in the logic, and not just repeat the same points already covered (we'd probably want to couple with more aggressive moderation). This could be automated via bot but we can start out doing it manually.

Some related questions to develop out this idea:

• Can we create a "template" to help in writing summaries?
• What kind of summaries do we want? How "neutral" can/should they be?
• Can we write guidelines for how summaries should feel?
• How can we empower folks to do the more aggressive moderation of "repeated" comments?

[Manishearth]

I've been advocating for this before, so i'm very much +1 for this. Ideally I think the summary would be written by someone who has not been an active participant (when active participants have made summaries my experience is that it makes things worse), but may discuss things with active participants.

I don't necessarily think that a new RFC has to be created: we could also edit the summaries into the top comment with a nice banner. Sometimes creating a new RFC would make a lot of sense, though.

[Ixrec]

I do think that resetting the discussion to a "mostly clean slate" is a significant part of the potential value here, but maybe we could achieve that without copy-pasting the RFC. Maybe when a known-contentious RFC is created we could also immediately open a discussion issue, reserving the PR itself for procedural comments and copy editing and other nits, so when it's time for a summary we simply close discussion issue 1 and open discussion issue 2.

For an RFC that was surprisingly contentious, where everyone initially thought that a few comments on the PR itself would suffice, I could much more easily see a fresh RFC being the best option.

[crlf0710]

I'm imagining a "debate bot", that reserves a "summary reply" at the very beginning of the rfc thread. All other people just discuss by send commands to it, and it will automatically put text into the summary and maybe some number counting. Generally it's asking the people to express their points into the summary text directly instead of asking someone else to read all text and summarize.

[nrc]

I like this idea, but in order to make it work I think it is important to identify who will write the summaries and why they would want to (if summaries are late or non-existent, the idea falls apart; if writing summaries falls upon the same busy people in a team, then this is likely to cause a lot more work for the 'wrong' people).

When to do this also seems important - once discussion slows down? When there are a certain number of comments?

This could be automated via bot but we can start out doing it manually.

If we can automate some or all of the idea, I think it would be great. I'm wary of adding process which is difficult to automate and increases manual workload, especially for a task which is likely to be time-consuming and feel low impact.

[Manishearth]

I've written about this before, I previously proposed a group of facilitators with no direct team connections for doing this. They can also enforce (non-CoC) participation guidelines if we have them.

[XAMPPRocky]

If we can automate some or all of the idea, I think it would be great. I'm wary of adding process which is difficult to automate and increases manual workload, especially for a task which is likely to be time-consuming and feel low impact.

While ideally any workload would be automated, I'm not sure we should start out with that in mind. In my experience starting with automation front loads a lot of work that needs to be done before the actual process can begin. This pretty deflating to any new initiative.

I also find that when a bot is introduced, people justify extra complexity by the fact that the work is being done by a bot. I believe that the process should be as apparent and understandable to someone who's reading their first RFC as possible.

To this end I'd prefer if we tried to iterate on our process "on paper", by trying it out different manual processes. And if/when we get to the point where we have something that we're mostly happy with, then we should start work on automation. It'll be a lot easier to build automations with that work done, as we'll have a much clearer idea of what its exact scope should be.

---

I've written about this before, I previously proposed a group of facilitators with no direct team connections for doing this. They can also enforce (non-CoC) participation guidelines if we have them.

I like this idea, though my first thought is how we'd recruit for this position. It feel it would run into the "How do we recruit people for leadership?" issue that has come up a couple of times recently.

[Ixrec]

I feel like I'm missing something really obvious here, but since people keep saying so much about "automation" I kind of have to ask: how could summaries of complex technical discussions possibly be automated at all?

It's not a "summary" unless it's been rewritten in more concise terms with all of the confusions, incorrect information, resolved tangents and minor details removed, so that it conveys all of the same important information in a small fraction of the word count. Today's robots just can't do things like that.

[XAMPPRocky]

@Ixrec I'm assuming the automation here would be essentially the process around the summary (e.g. closing a PR, and opening a new one), not the summary itself.

[BatmanAoD]

I like this idea, though my first thought is how we'd recruit for this position.

As a data point, I messaged Manish expressing interest in such a role right after I read the blog post. It's possible that I'm uniquely enthusiastic about such work, but I hope not!

It feel it would run into the "How do we recruit people for leadership?" issue that has come up a couple of times recently.

Maybe, but there's at least less incentive for people to volunteer in order to gain undue influence than there could be with leadership positions, I think.

[Manishearth]

@xampprocky I don't really view this as leadership in the sense that it doesn't have the same problems when it comes to people signing up.

We might need some leaders for this team, and those should be picked from folks who we know are capable leaders, but the large swath of work here is not super different from triage. I see this team as an excellent opportunity for community members to grow into leaders though. You get to have a ringside seat for discussions and learn how people interact, and also you get to make pretty high impact contributions.

Not to say that we shouldn't be careful (I'd be very wary of anyone who has had moderation problems being in this role) but with decent guidance I think this can work!

[Manishearth]

@lxrec Erin gave an example, but another example of automation would be rfcbot having a timer or heuristics to lock the PR periodically (either at comment count or time thresholds), requiring a facilitator to do one of these things (with input from the relevant team)

• update summary and declare summary done, unlocking the thread
• update with a summary and request the RFC be opened afresh, with the summary comment copied and perhaps some changes made
• decide a summary is unnecessary and ask the bot to unlock

Doesn't have to work that way, but it's an example

[nrc]

I don't really view this as leadership in the sense that it doesn't have the same problems when it comes to people signing up.

It's not leadership, but it is a position with significant power.

how could summaries of complex technical discussions possibly be automated at all?

Like others have mentioned, tools to help, rather than do the actual summary. I was also thinking that there might be alternatives to summaries which are easier to automate, I'm not sure what they are though.

While ideally any workload would be automated, I'm not sure we should start out with that in mind

Seems fine to start without automation, but I think the more an idea can be automated, the better it will scale. Ideas which add to folks' workload (especially linearly to the amount of discussion) shouldn't be written off, but it is a disadvantage.

[Manishearth]

It's not leadership, but it is a position with significant power.

True, but if it's a team that power can have oversight through its leaders. The main issue with leadership positions is that they're a lot of power with limited direct oversight.

[nikomatsakis]

Hmm, my mental model might be somewhat different than others here. I guess I suspect that working through this process would help.

It seems like the idea of a facilitator is to try and produce a kind of "neutral" summary. I'm not sure that's entirely possible or even necessarily. I feel like often there's a group that's "driving" the RFC or proposal, and I think what's most needed is to get a view into that group's thinking. i.e, what I'd like to see is something like a "formal response period" --

• where the feedback is reviewed, adjustments are made, and we issue a formal response with the new arguments etc.

I'm concerned that facilitators cannot (and, somehow, should not) produce those responses. That said, in earlier phases of the discussion (e.g., when brainstorming, etc), facilitators might have a pretty useful role. They could also be kind of "moderating" in the sense of "summarizing the concerns that will be responded to", I suppose, and other greasing the process.

I do agree with @nrc that there is a danger of conversation stalling out -- and of course that happens today all the time. That seems to me to be somewhat ok, but the only real fix for that which I see is to monitor the stage0 process, such that we only have RFCs that are being actively pursued by some team/group and where progress is being monitored. I think part of the problem here is that we currently lack that "top down driving".

[XAMPPRocky]

We should probably take a look at "GitHub Discussions", which is a new feature in beta available on some repos that allows you to essentially have a discourse forum embedded in a repository. To see if that helps address this issue, or changes how we would solve it. The main interesting thing is that in discussions every comment is threadable, which people are using for RFCs (e.g. a next.js RFC). We can probably reach out to GitHub have this enabled on one or two repositories to test it out.