Enumerate the requirements for wide review

[nigelmegitt]

I've observed that there is confusion about what constitutes a good wide review. It would be helpful to clarify in the Process the minimal requirements for processing review feedback, for example to match the good practice that was embedded in the old disposition of comments tool, namely:

• Announce by public notification and external liaison that must include the relevant groups listed in the Charter:
  • the review period,
  • the scope of the review being requested, and
  • the process for providing review feedback.
• Agree a WG disposition for each comment
• Send the WG disposition to the commenter (can have a latest response deadline not sooner than 2 weeks later)
• Record the commenter's response if any within the deadline

This is minimal - obviously groups that enter into discussions with the commenter and can document a conversation in which the WG and the commenter arrive at a mutually acceptable conclusion would satisfy these requirements.

[nrooney]

Read and understood. Will discuss at next CG meeting.

[dwsinger]

taking up, and we should clarify if this is practice or process

[jeffjaffe]

It is certainly a problem that chairs do not know what the minimum requirements are. In the past, people have wanted this treated as a practice rather than a process so that the Director might apply judgment in individual cases. But if chairs are totally confused, it might make sense to make this into a process issue.

[nigelmegitt]

By the way, since I raised this, it's become clear that horizontal review is considered to be a subset of wide review, so that needs to be brought into scope.

Additionally, I think that other W3C groups that are listed in the WG's Charter as dependencies also need to be notified.

One more point: there's some confusion about whether wide review is needed to enter CR, to exit CR, or both. The usual practice and expectation is that it is required in order to enter CR, but the Process clearly states that it is also required in order to enter PR. The ambiguity here is whether or not the same wide review can be used for entering CR and PR; since a spec cannot get to PR except from a CR, by definition all specs must have demonstrated wide review already. So why require it again explicitly for PR? Is it supposed to be a wide review of any deltas between the intended PR and the CR?

[dwsinger]

I think concrete suggestions to help clarify would be very welcome.

[LJWatson]

I think Section 6.2.3.1 has most of the information, but it's just not written in a way that's easy to parse. The broad topics I think this section needs to cover, are:

• What is a wide review/why is it sought
• When should wide review happen
• Who should be invited to participate/how should they be notified
• What should be done with the resulting feedback

If we can get some agreement around the high level points, I'll take a run at some words.

[nigelmegitt]

@LJWatson Yes, and I'd expand "what should be done with the resulting feedback" a bit to include the follow-up step with the commenter, in other words:

• how can the end of wide review be identified?

and add:

• What information about the wide review needs to be reported?

I don't agree that Section 6.2.3.1 has most of the information. That section currently describes the objectives of the Wide Review, which is definitely worth doing, but it does not address your summary bullet points. I think the edit needed is mainly additive rather than a change to the existing words.

[dwsinger]

I agree, the distinction between the review per se, and the consequent discussion, and possible action, may need to be clearer. The discussions can be effectively interminable; but the group needs to get past the milestone. Advice on how to handle that would be good.

[michaelchampion]

I don't know where to draw the line between formal Process and guidance on how to implement the policy in a way that will satisfy the Director. But from watching the pain chairs go through, it seems to me that wide review needs to be less labor intensive to track and quicker to conclude. That said, "wide review" is one of the features that distinguish W3C Recommendations from lighter-way alternative ways to define specs, so finding the right balance is critical.

Things I'd like to see in the process or guidance:

• Shipping is a feature! Accept that even Recommendations are not final; review feedback that doesn't get incorporated into this release can be considered in the next, so don't delay publishing a good spec in hopes of publishing a perfect spec.
• Feedback goes to where the chairs/editors say it should go, not to where the reviewers prefer. In practice that probably means a single GitHub issues list for the spec being reviewed, not the mailing lists or issues lists of the various horizontal groups. -Editors do have some responsibility to explain how they responded to review feedback, but requirements requirement to contact the reviewers to ask whether they are satisfied with the response seems excessive. If reviewers don't like the disposition they can re-open an issue, escalate to the chairs/team contact/Director, file a formal appeal, etc.
• Each community is different, so I would give WG chairs/team contacts a fair amount of discretion on how to get useful reviews from the right experts.

[nigelmegitt]

Editors do have some responsibility to explain how they responded to review feedback

Why "Editors"? I think this is a WG responsibility.

requirement to contact the reviewers to ask whether they are satisfied with the response seems excessive. If reviewers don't like the disposition they can re-open an issue, escalate to the chairs/team contact/Director, file a formal appeal, etc.

@michaelchampion I disagree - it is too onerous on external individuals and organisations to force them around this loop, and it is courteous to conclude the conversation that they began. It is also useful information if they agreed or disagreed with the WG's disposition, since the Director can rapidly filter out the "agreed"s and focus only on the "disagreed"s which is where the attention is likely needed.

[michaelchampion]

The question I'm asking is whether we want to make it onerous for chairs / editors to process feedback at the cost of making the standards process slow and bureaucratic or to make it more onerous for external reviewers to track the resolution of their issues and contest the WG / editor resolution.

I see a lot of work that once might have gone to W3C being done in WHATWG or open source projects that operate by "lazy consensus" http://en.osswiki.info/concepts/lazy_consensus. They give editors / committers clear ownership, and puts the burden on reviewers who don't agree to speak up and make their case to do something different.

But I agree it's a difficult balance to strike -- a value of W3C standards over those from some more agile organizations is a broad base of stakeholders who have given specs wide review. High quality input from external experts who will engage to make their case is definitely valuable, but such reviewers are generally happy to engage directly in the WG's issues list in my experience.

[dwsinger]

We want good results without it being onerous (indefinite amounts of effort) for anyone, or taking indefinite amounts of time. We have to find a better way to do this.

On Feb 28, 2018, at 8:53 , Michael Champion notifications@github.com wrote:

The question I'm asking is whether we want to make it onerous for chairs / editors to process feedback at the cost of making the standards process slow and bureaucratic or to make it more onerous for external reviewers to track the resolution of their issues and contest the WG / editor resolution.

David Singer Manager, Software Standards, Apple Inc.

[nigelmegitt]

High quality input from external experts who will engage to make their case is definitely valuable, but such reviewers are generally happy to engage directly in the WG's issues list in my experience.

@michaelchampion Sometimes they do, but often they do not: a lot of feedback comes my way from other standards organisations representing a constituency who are giving collective feedback with the goal of adopting/referencing W3C standards within other specifications. Their preferred engagement model is much more ponderous, and consists of liaison messages. Obviously where possible we do try to engage more directly, but members of other standards organisations do not always want to come out in the open.

[LJWatson]

I've taken a first pass at some words. My hunch is that this is guidance rather than process, even assuming it captures what we hope it will. In the interests of having something concrete to work with though...

the purpose of a wide review is to seek feedback on a document from the entire web community, including the general public. The goal is to make the document as robust as possible before it becomes a Recommendation.

A wide review can be conducted at different stages of a document's maturity. Depending on the reason for the wide review, the set of stakeholders may vary.

As a minimum, a comprehensive wide review should be conducted as the document prepares to transition to Candidate Recommendation. At this stage a wide review request should be sent to the following stakeholders:

• The Working Group responsible for the document
• The liaison Working Groups documented in the Working Group's charter
• Other?

In addition, a notice should be sent to [insert email here].

A wide review can also be conducted at the following times:

• When the First Public Working Draft is published
• Whenever an updated Working Draft is published
• Other?

To facilitate wide review, a document should have a log of changes made since its previous publication on TR.

Issues raised as a consequence of a wide review should be filed on the Github repository for the document. This streamlines the management of issue discussion and resolution, and makes sure the person or organisation that files the issue is informed of its progress.

[dwsinger]

It's a start, but I think the hard questions are around "what is a review? when is it complete? what level of reaction is needed? how much response is needed? how much consensus with the commenter is needed? can review to-and-fro be allowed to drag on forever? can deadlines be put on the stages? how much documentation trail is needed? are tagged GH issues good enough?"

[nigelmegitt]

The Process shouldn't say where issues should be raised because it's a WG decision about how to work and what tools to use. However the SOTD in the document being reviewed must do so I think. If it's not already a requirement then it needs to be made one.

The "other" stakeholders need to include any external liaisons registered as having an interest in the topic (see Liaisons for the list) or listed in the WG's Charter, and the general public, as notified minimally on the w3.org homepage but also by any other W3 communications channel such as twitter, RSS etc.

I'm slightly troubled by the idea of differing sets of stakeholders for wide review depending on maturity level. It would be simpler to have one process that is wide review, and reuse it wherever needed.

For what it's worth, my pet name for a review process that only goes to other W3 groups is "narrow review".

[dwsinger]

Team is working on this and we expect a proposal

[wseltzer]

@swickr can you share updates from your conversations with i18n and a11y?

[dwsinger]

One of the bad aspects of wide, and particularly cross-functional, review, is that when voting at the AC I actually have no idea whether the TAG, i18n, accessibility, privacy or security experts 'sign off' on the spec., or think it should not be published in its current form. I don't even get that one bit of advice. Getting 'one bit' from the world at large is harder, but we could be clearer about x-func.

[frivoal]

A disposition of comment tends to give you some of that information, although it is a little hard to extract the answer to that particular quesiton from it. It tells you whether lots of people have filled comments about the spec, and whether their issues have been addressed. If there are indeed many, and they include the various cross-functional review groups, and the comments have been addressed to their satisfaction, then it is reasonable to infer that they are OK with the spec. But that's pretty indirect.

An explicit process for signing off is would introduce overhead, which people would then complain about and use to call for not using the Process at all. I wonder if we can do something about the presentation of the information we already have to make this easier for AC reps to review.

We already ask in about wide review in the when we make people file transition requests

We could expand this a little to explicitly ask whether each of the cross functional review groups has been looped in (and evidence that it has). Statements like the following don't require blocking on sign off, but would still be useful.

i18n reviewers have been informed about this specification, and have no standing concern about it. Link link link link.

---

Tangential issue: This made me think about, and file https://github.com/w3c/w3process/issues/260

[dwsinger]

Yes, I envisage a simple green/yellow/red light from each x-func group: green -- we see no reason why the spec. shouldn't proceed yellow -- we have concerns and the AC and Director should weigh them carefully red -- we don't think this should proceed in its current form

obviously, for yellow and red it should be easy to find out what the concerns are. Maybe this meshes with your form in #260

[LJWatson]

As an aside: I've been talking to chairs, team contacts, editors, and others, about horizontal review (as an AB priority task). All things being equal, I'll have finished this information gathering exercise in time to report to the AC in April.

I don't want to preempt anything, but it already seems clear that more clarification, both within the Process and without, will be a clear recommendation.

[LJWatson]

Looking at the recommendations from the AB priority task to review Horizontal Review (HR), and at the request in #264, we definitely need greater clarity.

We need to better define HR. HR groups are mentioned twice in the Process (in 5.2.6 and 6.2.3.1), and in both instances link out to (dated?) information on /Guide.

We need to better define when HR is required; either as part of Wide Review (WR) or as an independent function. The mention of "28 days" in 6.2.3.1 as an appropriate notice time for WR before CR is not helpful.

We should also be clear about what HR evidence is required before a spec can transition (notably to CR).

This need not all be done in the Process itself, though I think that better definition of HR (or at least the horizontals that must be consulted), and the expected timing of HR should be.

We could drop mention of Horizontal Review from section 6.2.3.1, and add a new section into 6.2.3. Here is a strawman attempt at some words:

Section 6.2.3.2 Horizontal Review

The objective of Horizontal Review is to make sure W3C specifications are robust in terms of accessibility, internationalisation, privacy, security, and technical architecture.

Horizontal Review must be solicited from the Horizontal Review groups at the following points of maturity:

• Within 28 days of publication of a First Public Working Draft
• At least 60 days before requesting transition to Candidate Recommendation

Horizontal Review of all new and significantly revised sections of a specification should also be sought from the Horizontal Review groups.

The Director will consider evidence of Horizontal Review before approving transitions. Appropriate evidence may include:

• Documented Horizontal Review requests for each of the Horizontal Review groups, within the expected time-frame.
• Documented response from each of the Horizontal Review groups, within the expected time-frame.
• Documented tracking of Horizontal Review comments and Working Group responses.

OK, have at it... :)

[nrooney]

On 19 Apr 2019, at 13:16, Léonie Watson notifications@github.com<mailto:notifications@github.com> wrote:

Section 6.2.3.2 Horizontal Review

The objective of Horizontal Review is to make sure W3C specifications are robust in terms of accessibility, internationalisation, privacy, security, and technical architecture.

Horizontal Review must be solicited from the Horizontal Review groupshttps://www.w3.org/Guide/process/charter.html#horizontal-review at the following points of maturity:

• Within 28 days of publication of a First Public Working Draft
• At least 60 days before requesting transition to Candidate Recommendation

Horizontal Review of all new and significantly revised sections of a specification should also be sought from the Horizontal Review groupshttps://www.w3.org/Guide/process/charter.html#horizontal-review.

The Director will consider evidence of Horizontal Review before approving transitions. Appropriate evidence may include:

• Documented Horizontal Review requests for each of the Horizontal Review groupshttps://www.w3.org/Guide/process/charter.html#horizontal-review, within the expected time-frame.
• Documented response from each of the Horizontal Review groupshttps://www.w3.org/Guide/process/charter.html#horizontal-review, within the expected time-frame.
• Documented tracking of Horizontal Review comments and Working Group responses.

OK, have at it... :)

I like it!

I was going to mention something about the questionnaires, but I think your “documented requests” and “documented response” allows for flexibility whilst making sure the HR is still recorded. No changes from me!

.

[r12a]

Horizontal Review must be solicited from the Horizontal Review groupshttps://www.w3.org/Guide/process/charter.html#horizontal-review at the following points of maturity:

• Within 28 days of publication of a First Public Working Draft

Thanks for your proposals @LJWatson. Many years ago, over a period of about 2-3 years, i was part of a small task force working on exactly this problem for internationalisation of the corporate engineering process at Xerox, and strengthening the case for HR around the time of FPWD is very much in line with the Quality Assurance process we followed then. If major concerns are to be raised about a design approach, this is the time when there is flexibility and time to react, and the cost of change is lowest.

I have a few concerns and suggestions, however, which i'll list below for discussion.

[1] There is no gating requirement for FPWD HReview, and i imagine therefore that many groups will not do the review. There may be no way to find out whether they did it, nor any way to address substantive directional issues until it becomes very late in the process (which is problematic for all concerned).

[2] It would be ideal to require evidence of HReview as a prerequisite for transitioning to FPWD. The person approving the transition to FPWD could ask for evidence that it is done before approving the transition. However, i think that the way i would prefer this to happen is for the group to do a self-review and get it checked by the HR group. This builds understanding into the WG, at the same time as reducing bandwidth issues for the HR group, and avoiding the likelihood of a bottleneck in busy periods. (For i18n, this would mean the WG starting with this short checklist and following links where appropriate to the more detailed checklist.) Perhaps we could change the wording to require evidence of self-review for horizontal issues prior to FPWD, and require acknowledgement by the HR group within the following 28 days??

[3] Another significant help could be to assign one or more members of the WG to the role of Review Champion (or some such name). That person wouldn't need to know all the answers (though they would probably learn a lot from the experience, which they can then pass on to other groups later), but would be expected to (a) ensure that horizontal reviews take place in a timely fashion, (b) be generally aware of areas that may need attention as the work proceeds (see the short checklist above), and (c) facilitate contact with the horizontal groups at appropriate points for advice. (This role could actually be expanded to cover wide review as well.)

hth

[chaals]

@r12a wrote:

[1] There is no gating requirement for FPWD HReview, and i imagine therefore that many groups will not do the review. There may be no way to find out whether they did it, nor any way to address substantive directional issues until it becomes very late in the process (which is problematic for all concerned).

[2] It would be ideal to require evidence of HReview as a prerequisite for transitioning to FPWD. The person approving the transition to FPWD could ask for evidence that it is done before approving the transition. However, i think that the way i would prefer this to happen is for the group to do a self-review and get it checked by the HR group. [...]

I agree that there should be a requirement to demonstrate that the first "horizontal functionality" reviews - a11y, architectural coherence, i18n, privacy, and security have been considered in order to reach FPWD.

I am increasingly of the view that it should be a more stringent requirement at least in terms of self-review, identifying a champion, and having the relevant group look over the initial approach. This is especially the case where a group has "incubated" their proposed work before even considering a request for FPWD.

The main purpose of dropping Last Call, in Process 2014 which also had requirements for regular WD heartbeats, was to push for a more continuous and granular review process in line with the way that features are specified and implemented in reality, as an attempt to disrupt the current common failure mode of getting to the imagined end of the process only to discover that a group has made say i18n almost impossible to implement successfully, or there is a grave security hole in the fundamental design of a feature.

[frivoal]

I agree that around FPWD is a good time for horizontal review. I am unsure about making getting and documenting horizontal review as a gate-keeping function though. Instead of increasing the amount of horizontal reviews that get made, it could well decrease the number of specs that enter the REC track at all, and push them to alternative processes (WHATWG, Evergreen, Incubation-forever...) with less perceived bureaucracy.

Instead, how about making sure that publication as FPWD automatically triggers a request for review to all the relevant groups? In process terms, that can be framed as something the team must do, but it practice maybe it could be automatically sent.

[r12a]

Instead, how about making sure that publication as FPWD automatically triggers a request for review to all the relevant groups? In process terms, that can be framed as something the team must do, but it practice maybe it could be automatically sent.

@frivoal i believe this suggestion misses an important part of my recommendation above. I'm recommending that the WG developing the spec self-review their document. I think that's reasonable. The WG needs to take responsibility for addressing horizontal concerns as part of their design, and they will better internalise useful knowledge that way. It will also help to significantly reduce the bottleneck as horizontal groups receive more requests for review than before. I envisage self-review as much more efficient than relying (solely) on an external group for review, since the horizontal group is typically unfamiliar with the technology, the features, and the spec, and already has limited bandwidth when it comes to scheduling the reviews, and following a formal and time-consuming process to submit comments.

I'd also be disappointed if there was a reliance on staff, rather than on horizontal champions within a WG, for triggering review requests, since again it seems like an abdication of responsibility.

Let's also be clear: if people want to move to the alternative processes you mention, they would still be well-advised to address horizontal issues early on, because otherwise they are only delaying the inevitable and in the meantime building up costs for any rework needed later.

[LJWatson]

@R12A commented:

[2] It would be ideal to require evidence of HReview as a prerequisite for transitioning to FPWD. The person approving the transition to FPWD could ask for evidence that it is done before approving the transition. However, i think that the way i would prefer this to happen is for the group to do a self-review and get it checked by the HR group. This builds understanding into the WG, at the same time as reducing bandwidth issues for the HR group, and avoiding the likelihood of a bottleneck in busy periods. (For i18n, this would mean the WG starting with this short checklist and following links where appropriate to the more detailed checklist.) Perhaps we could change the wording to require evidence of self-review for horizontal issues prior to FPWD, and require acknowledgement by the HR group within the following 28 days??>

I like the idea of a self-review before requesting transition to FPWD, and the corresponding response timeline.

Where does the TAG design review fit into this? My understanding from @Torgo, is that the TAG design reviews are intended to happen at some point in the early planning stages of a proposed specification. I know PING and TAG have developed a short privacy checklist for use during TAG design reviews. Is the I18n checklist you mention to be used during TAG design reviews also?

@frivoal commented:

Instead of increasing the amount of horizontal reviews that get made, it could well decrease the number of specs that enter the REC track at all, and push them to alternative processes (WHATWG, Evergreen, Incubation-forever...>

The Process (6.2.3.1) already says that WR including HR is expected at each major transition, and that the Director will consider evidence to that effect before approving the transition. This proposal isn't trying to change that, but to provide the clarity that almost everyone I spoke to during the AB review felt was needed.

I'm not sure there is evidence to suggest clarifying HR requirements would push people away? If there is, there is also the argument that these things are important enough to take that risk.

I don't think the provision of evidence is hard. From experience on either end of the HR process (as part of APA WG and co-chair of WebPlat WG), it simply requires making a formal request, following up on that request, responding to and/or addressing the issues that were raised in a timely fashion, and in a place that can be publicly accessed.

@R12A also commented:

[3] Another significant help could be to assign one or more members of the WG to the role of Review Champion (or some such name). That person wouldn't need to know all the answers (though they would probably learn a lot from the experience, which they can then pass on to other groups later), but would be expected to (a) ensure that horizontal reviews take place in a timely fashion, (b) be generally aware of areas that may need attention as the work proceeds (see the short checklist above), and (c) facilitate contact with the horizontal groups at appropriate points for advice. (This role could actually be expanded to cover wide review as well.)>

This was one of the recommendations from the AB review. It's something I've seen work in other environments, and if W3C does move to an Evergreen model, it will be come even more important.

I gather WAI tried to do this in the (distant) past, and it didn't work out. My hunch is that it was due to a lack of bodies. There are many more accessibility professionals these days, and I even think we might see more participation from people if they were able to participate in the specification WG itself - this is certainly something we pushed for during my time at TPG.

[michaelchampion]

I like the direction this thread is going: instead of seeing HR as something done by external people late in the process, ensure that WGs have built a community with the various horizontal competencies early in the process. "Competence" doesn't have to mean "expertise", but it does entail having someone designated to "own" the responsibility for liaison with experts, filling out self-assessments, resolving issues, etc. I'd go so far as to say it's a precondition to chartering a WG to having built up real relationships with the various horizontal communities, as opposed to typical practice of just listing them in the semi-boilerplate liaisons section of a charter. Then each subsequent maturity stage would require higher and higher levels of proof that the horizontal considerations have been adequately considered.

[nigelmegitt]

Issue divergence alert: this issue was raised about Wide Review in general; Horizontal Review forms part of that, but making improvements to HR alone won't tackle this issue. It'd probably be a good idea to factor out the HR stuff into a separate issue.

[LJWatson]

nigelmegitt commented:

It'd probably be a good idea to factor out the HR stuff into a separate issue.>

I'm hesitant to do this because HR is part of wide review, so if we want to better explain the requirements of wide review, then discussing HR in the course of that makes sense to me.

Plus I'm not a fan of splitting discussions mid-thread :)

[LJWatson]

I've taken ideas from this issue and feedback from the AB priority task, and put together a proposed amendment to section 6.3.2.1 of the W3 Process.

It's in a wiki because it's just a straw proposal at this stage. It may not all be needed or wanted in the Process, though I think there is a strong case for it being there and for us to provide further implementation guidance in /guide or somewhere), and inevitably there is more discussion to be had on the whole thing anyway.

[chaals]

There is an assumption here that there are one or more distinct "wide reviews". That doesn't match the basic requirement of the Process, which is that there has been review by a sufficiently wide range of experts and interested parties to ensure there isn't some obvious flaw in the proposed technology.

The ideal situation is that as features are introduced they are assessed, and they are not finalised unless there is such review of each feature.

The pre-2014 process assumed this would be formalised by the "Last Call" stage of the process, but in practice that often meant that reviews were performed late in the development cycle. In many cases the outcome was that implementations would not be changed in accordance with the review outcomes, or only minimal changes would be made as a compromise solution. In cases where they were, it was an extremely disruptive and expensive point of the technology's lifecycle to make such changes.

[LJWatson]

@Chaals commented:

There is an assumption here that there are one or more distinct "wide reviews". That doesn't match the basic requirement of the Process, which is that there has been review by a sufficiently wide range of experts and interested parties to ensure there isn't some obvious flaw in the proposed technology.>

The Process already implies there should be multiple review points throughout the specification lifecycle, but by its own admission "The requirements for wide review are not precisely defined". The proposal attempts to provide that definition, in response to clear feedback from team contacts, chairs, and editors, that it was lacking.

The ideal situation is that as features are introduced they are assessed, and they are not finalised unless there is such review of each feature.>

The ideal hasn't changed. The proposal just recognises that the ideal isn't being met as consistently as it should be (because there is no clear definition of what is or isn't expected), and it provides a framework that aims to rectify that.

The pre-2014 process assumed this would be formalised by the "Last Call" stage of the process, but in practice that often meant that reviews were performed late in the development cycle. In many cases the outcome was that implementations would not be changed in accordance with the review outcomes, or only minimal changes would be made as a compromise solution. In cases where they were, it was an extremely disruptive and expensive point of the technology's lifecycle to make such changes.>

This is why the proposal recommends self-review before transition onto TR, requires review at FPWD and before CR, and recommends interim reviews for major updates.

The spirit of the Process hasn't changed in the proposal, the only change is the level of clarity around what wide review and horizontal review looks like in practice.

[dwsinger]

wrt https://www.w3.org/wiki/Wide_and_Horizontal_Review

1) I don't think we should insist on (mandate) a green light from all horizontal groups. Some will fail to vote, and a yellow light is "proceed with caution". I'm not even sure that a red light should be a mandated stop.

2) The lights should be expressed as the opinion of that group, so change to read:

Green: this groups sees no reasons the specification should not progress along the Recommendation Track Yellow: this group has issues that should be weighed carefully by the Director and AC before approving transition. Red: this group believes that this document should not progress as is, that it has issues the WG should address before progression is approved

i.e. green should not be read as a statement that the group supports the document, etc.

[dwsinger]

On Jun 25, 2019, at 5:48 , chaals notifications@github.com wrote:

There is an assumption here that there are one or more distinct "wide reviews”.

ah. I think there are two distinct shells of the onion here

wide review includes

• AC review
• team and Director review
• and cross-functional review
• external, public, review, which includes liaison review

I’d like to have the first two informed by the third, as at the moment, I think they get lost.

That doesn't match the basic requirement of the Process, which is that there has been review by a sufficiently wide range of experts and interested parties to ensure there isn't some obvious flaw in the proposed technology.

The ideal situation is that as features are introduced they are assessed, and they are not finalised unless there is such review of each feature.

The pre-2014 process assumed this would be formalised by the "Last Call" stage of the process, but in practice that often meant that reviews were performed late in the development cycle. In many cases the outcome was that implementations would not be changed in accordance with the review outcomes, or only minimal changes would be made as a compromise solution. In cases where they were, it was an extremely disruptive and expensive point of the technology's lifecycle to make such changes.

David Singer Manager, Software Standards, Apple Inc.

[michael-n-cooper]

Judy drew my attention to Léonie's proposal. I support the goal of systematically documenting horizontal review practices and expectations in the Process document, but am worried aspects of this proposal will cause significant push-back. I write this as someone who both performs and solicits horizontal review and see challenges from both perspectives.

Where it says "Horizontal review MUST also be solicited...", I think that is a good practice, but the "must" statement adds three months to the Rec track process, which I think will concern spec developers. In principle those can be done in parallel with other activities; in practice, that may be a challenge, particularly soliciting horizontal review 2 months before CR is predicted. The readiness of a spec approaching CR transition can be difficult to predict, unless the group uses a timeline-centric work mode where timeline trumps readiness. There's also the possibility that a group could think it's close to CR and issue the horizontal review notice, then get a lot of comments, spend a year making substantial changes, and then want to go to CR without redoing the horizontal review on the basis that it was already done, yet the set of accumulated changes really means a new one is needed. Also I don't see a need to allow 28 days after FPWD to solicit horizontal review, I think it's reasonable to require a horizontal review notice go out with the FPWD publication announcements.

The simplest way I can think of to address this the proposal is to change the quoted sentence and subsequent bullets to read:

Horizontal review SHOULD also be solicited from the Horizontal Review groups at the following points of maturity: - Upon publication of a First Public Working Draft; - Approximately 60 days before requesting transition to Candidate Recommendation.

For this to achieve the desired benefits for horizontal review, the expectations of the Director at transition should be strengthened. I don't have great wording, but maybe changing the sentence beginning "The Director will consider..." and the subsequent bullets to read:

The Director MUST consider evidence of timely solicitation of horizontal review before approving transitions. Appropriate evidence may include: - Documented requests for horizontal review issued and appropriate points of the specification maturity and with reasonable timelines for response; - Documented response from each horizontal review group, when received; - Documented resolution of issues raised during horizontal review.

Note this wording isn't intended to constrain the Director from approving transitions even with weak horizontal review, if there's a compelling reason for it, but to ensure the Director examines the question and to underscore to groups that their best chance of success with the transition is to issue timeline review requests.

Another concern I have is that the current wording requires that all horizontal review groups make an affirmative reply on all specs. That allows horizontal review groups to potentially stall a spec's advancement through simple inaction, which won't be a popular outcome. It also creates a big burden on horizontal review groups to respond to many specs in a timely manner on an ongoing basis. In the APA group, it takes time for us to organize reviews and gain consensus on formal comments, and it's best for us to focus on the specs that we feel need the attention. Even a simple "no comments from us" response is hard to make as it requires affirmative effort. We expect that if we don't comment a group will proceed, and the proposal does set reasonable timelines for that window, so it's easier to allow that to happen for low-concern specs.

To address this, I suggest removing the bullet "Received a "green light" from each of the horizontal review groups.".

Final comment, I'm not seeing value in setting a Horizontal Review Interest Group into the Process, and in particular don't think such a group should be invite-only. I think a simple mailing list should be sufficient, populated by anyone who thinks they have a role to play, though an IG is ok if not invite-only. We might want to consider subscribing a nominated list from each horizontal review group as well to ensure groups will see notifications, but maybe that could be left to an implementation detail. Similarly, the "traffic light" response categorization might be a useful concept ("good to go", "minor tweaks", and "major comments") but not sure it needs to be at the Process level. So the entire section of the proposal with the heading "Horizontal Review Interest Group" should be moved to a supporting resource with tweaks.

[dwsinger]

I agree, I thought we were looking for ways to make wide and specifically x-func review easier and more effective, not more coded into rules in the process. My traffic light idea was intended to make it easier for the x-func groups to summarize what they saw and for the AC and Team+Director to see those summaries.

I really like the idea that we tell groups that they should alert their team member 60 days before an expected transition, and at that point we put up the WBS ballot for teh x-func group reps to respond to, and when the ballot goes to the AC, they and Team+Director are explicitly linked to the Results page of the ballot. I'm guessing this ballot should have a minimum duration (30 days?) which might delay the transition request if the WG doesn't ask in time.

I'd like to run this experimentally first, with nothing, or as little as possible, codified into the Process.

[swickr]

@LJWatson wrote:

The spirit of the Process hasn't changed in the proposal, the only change is the level of clarity around what wide review and horizontal review looks like in practice.

and much earlier @dwsinger commented:

I think the hard questions are around "what is a review? when is it complete? what level of reaction is needed? how much response is needed? how much consensus with the commenter is needed? can review to-and-fro be allowed to drag on forever? can deadlines be put on the stages? how much documentation trail is needed? are tagged GH issues good enough?"

The draft proposal adds specification of when and how (the mechanics) but not what (clarity on the goals). The first two are certainly important but to Chaals' point the value-add that the W3C Process should bring is in emphasizing what we want to achieve through these reviews.

Reviews should be solicited widely, begun early in the design cycle, be as specific as the reviewer can be (based on the material available for review), be responded to in a timely manner with reasonable deliberation, be repeated often as the design matures, and have the purpose of allowing the communities of expertise to iterate on problems, requirements, and solutions.

Some of the current frustration with our processes is the absence of consistent mechanics. Let's keep the bulk of the mechanics in the Guide Book where we can capture and refine current best practices. Let's improve the text in the Process to address goal-related uncertainties that Léonie heard in her interviews of chairs, team contacts, and editors:

• [Horizontal] Review is required
• Working Groups must solicit reviews in a timely manner
• Working Groups should solicit reviews frequently when there are material changes to the design
• Working Groups must respond to review comments in a manner that allows the conversation to continue
• Working Groups can and should facilitate review by
  • producing explainers to accompany their specifications
  • soliciting initial review no later than FPWD
  • using self-review checklists where available
  • maintaining human-readable changelogs
  • maintaining a discussion record for each individual review comment

[brewerj]

+1 to Michael's specific concerns about how this might turn out in practice;

+1 to Ralph's highlighting that a key issue is the lack of consistent mechanics for reviews; but Ralph in your making mostly very generalized statements, I wonder if we would get any more consistent results.

David, an experiment might indeed be useful, but I would recommend doing so only after it has stabilized more in response to feedback.

[frivoal]

@chaals said

There is an assumption here that there are one or more distinct "wide reviews". That doesn't match the basic requirement of the Process, which is that there has been review by a sufficiently wide range of experts and interested parties to ensure there isn't some obvious flaw in the proposed technology.

@LJWatson responded

The Process already implies there should be multiple review points throughout the specification lifecycle

I'm with @chaals here: to my reading, the process does not call for multiple rounds of a time-boxed activity called a wide review. It calls for the document to be continuously reviewed by multiple parties (including HR groups and the general public), and at multiple points, we check that the review has been wide enough. This isn't in contraction with calling for reviews from specific groups at specific points, but that's an activity in support of the wideness and thoroughness of the review. That isn't on its own a wide review. The draft in the wiki takes this in a different directly, and speaks of "a wide review" as a thing that happens several times and defined points in the document's evolution.

I think the proposals form @michael-n-cooper and @swickr (or some blend thereof) are much more in line with what I'd expect to see in the process. After that, I think the bulk of the investment should be in better tooling. On that topic, I've seen so far two main axes proposed, both of which seem worth pursuing:

• generalize and improve the tools from the i18n community to flag individual issues, in order to improve coordination between HR groups and WGs on individual issues on an ongoing basis.
• set up some kind of traffic-light signaling mechanism allowing HR groups to communicate their satisfaction with a spec as a whole

Further, I think the way WGs announce their publications varies a lot, and that this damages the visibility of spec updates: there's public-review-announce@w3.org, https://www.w3.org/blog/news/, WG specific mailing lists(s), blog(s), and twitter account(s)... all of that is handled manually, with varying degrees of consistency. I've been thinking that we should try and consolidate that, probably by being presented (by echidna?) with a form when you try to publish, where you have to fill in enough details (link to the spec, abstract (auto-filled form the spec), link to explainer, link to changelog, summary of what is important to review this time, where to file issues) for all these announcements to be generated automatically (and sent to whichever place is relevant for that particular WG/spec). Included in that would be some check boxes that let you forward the announcement to the various HR groups (mandatory for FPWD, opt-in for subsequent publications). Making it easy to ping HR groups and the general public in a consistent manner should help ensuring it happens more frequently. Right now it take quite a bit of manual peeking around the w3c website to figure out even how to send requests for review to horizontal groups. If this could be replaced by a single checkbox on the path to publication saying "yes, now is a good time", I bet it would happen a lot more often.

[r12a]

We seem to keep getting caught up in endless discussions about where things should be said and exactly how things should be worded. Please permit me to have a small rant.

I looked back over the review requests i18n has had in the past 6 months (which is representative of previous years) and i see that all but one requests were sent two, or with luck three, weeks before a fixed date for transition to CR. The LC process wasn't good, but this is worse!

(Btw, the one exception was from another group that does horizontal reviews.)

It results in the i18n group having to immediately push aside all the other things they are working on and expend significant amounts of stress and good will contributions to schedule discussions, understand the technology, and negotiate often significant changes with the Working Group that produced the spec, and all at the very last minute. Frankly, in those timescales we can't be expected to do a proper job.

For me, the key problem is that WGs don't seem to be getting the message, despite documentation having existed for some time in various places. It seems too easy for them to ignore or be unaware of what they should do. If we're going to fix this we need to require actionand review at specific points and, most important for this discussion thread, we need to find a way to make the information unmissable for the WGs. Those, to me, are the fundamental issues we should be discussing here. That's why i think we need to change the Process Document, and change the way transition reviews happen.

And i hope that we can make this happen much sooner than later.

Here are my comments on the document and the foregoing related to Horizontal Reviews. I think...

1. Leonie's proposal to have a single mailing list to which groups are required to send review requests is brilliant. I think we can do that right now, and i volunteer to set it up, but we need to require WGs to use it. How do we make sure that will happen? The Process Doc seems the right place to me.
2. horizontal-review-announce@w3.org seems like a good name for [HR IG email].
3. like, Michael, i'm not sure we need to build an IG around that mailing list. At least not initially.
4. We also need to make it clear (as Léonie's document does, and unmissable what information to send with a review request. How do we do that? My guess is that The Guide is appropriate. I think a notification request should also indicate which HR groups a review is being requested from.
5. we need some convincing way to get WGs submitting wide review requests at least 60 days before CR. How do we do that? For me, this has to be in the Process Doc. It's not as if we aren't saying that in other places, but WGs are not taking notice. I'd also be happy to see a mention in the group charter. If you disagree with putting this in the Process Doc, please explain what you think will work - it's already mentioned in the Guide, at the Wide and Horizontal Review page, and in all the i18n pages related to review, and we've mentioned it in AC meetings, but it's not making any difference.
6. SELF-REVIEWS, SELF-REVIEWS, SELF-REVIEWS! A lot of the pressure on HR groups could be relieved, and the general quality of work at W3C will be leavened, if groups self-review their work from time to time. My view is that we MUST require this prior to transitioning to the Rec track (ie. FPWD for Rec docs), and suggest that the WG repeats it at other times. We already have several checklists, right now, that groups can use. How do we get them to do it? . See the next point, but also i think we should have a dedicated section (to make it more visible) at https://www.w3.org/wiki/DocumentReview#Horizontal_Groups. We should also recommend that the WG do a self-review prior to sending a review request to the HR group (particularly the 60 day prior to CR review). Perhaps we could also have a session at TPAC that explains how a self-review works, and the practical steps involved.
7. it MUST be a condition of transition to the Rec track (ie. prior to FPWD for Rec docs) that (a) the WG has done self-reviews in the horizontal areas for which such are available, and (b) that they send notification to the review request mailing list. How do we make sure they will do that? I'd prefer the Process Doc to require it, but at least we need to bake it into the formal review process, and let WGs know we did that.
8. i don't think WGs have to have engaged with the HR group in review at FPWD stage, unless the HR group requests it, and so i don't think a green light is needed (for FPWD). (We don't have the bandwidth to review all, but we do want to be given the chance.) (It may be easier to write the text in Léonie's doc by separating FPWD from 60-day-prior/other reviews.)
9. whoever is conducting the CR transition review MUST be shown evidence that HR has taken place, and the issues have been dealt with
10. WGs SHOULD be encouraged to nominate an HR champion, whose job is to liaise with HR groups, monitor timelines, manage self-reviews, and (only if experience allows) raise potential HR issues for discussion with the group. Should this be added to the Process Doc? I'd say the Guide may be sufficient, but the evidence doesn't seem compelling that WGs will take notice...
11. i think Florian has some good ideas for tooling improvements, but let's please not delay the implementation of the change by waiting for the tools. They should just make it easier, once they come along.

[frivoal]

i think Florian has some good ideas for tooling improvements, but let's please not delay the implementation of the change by waiting for the tools. They should just make it easier, once they come along.

While I think we agree on the what is the ideal state of the world, this is where we differ. If we put lots of MUSTs and mandatory review periods in the process without improving the tools, we're more likely to drive people off the REC track entirely (or stay in WD stage forever) than to make them well behaved.

As far as the Process itself is concerned, I am not comfortable going much further than then proposal in https://github.com/w3c/w3process/issues/130#issuecomment-505580986 until we have made it easy to comply with the new requirements.

[frivoal]

we need to find a way to make the information unmissable for the WGs.

That, I totally agree with. But I don't think the Process is a good place to do that.

[r12a]

If we put lots of MUSTs and mandatory review periods in the process without improving the tools, we're more likely to drive people off the REC track entirely (or stay in WD stage forever) than to make them well behaved.

But i (and i think Michael too) are not asking for lots of MUSTs and mandatory review periods. Basically what i'm asking for is:

1. do a self-review before publishing FPWD, and notify the HR groups when you publish
2. contact the HR groups for review at least 60 days before CR

That's it. These are things we already ask groups to do, and will continue to ask groups to do, i hope. For me, the Process Document describes the policy of the W3C wrt work processes, and i see those things as describing good policy. I agree that the mechanics can be dealt with elsewhere.

That, I totally agree with. But I don't think the Process is a good place to do that.

Presumably, the continuation of your argument is that tooling will make the difference? (Because, the other alternatives we've tried so far have clearly failed, and to be honest i don't see the recommendations in https://github.com/w3c/w3process/issues/130#issuecomment-505580986 as likely to change anything much.)

[LJWatson]

I've updated the proposal based on comments here. It would take too long to respond to everyone's comments individually, but hopefully the following amendments capture most if not all of it...

I've edited the information on wide review. I think @Chaals, @frivoal, and I might have been talking at cross-purposes. I agree that wide review is a holistic thing, but in attempting to clarify exactly what that means, it gave the impression of being multiple separate processes instead of multiple touch-points throughout a continuum. I've borrowed some of @rswick's words, and hopefully it's closer to the mark this time.

I've made it a MUST to send a notification to public-review@ on publication of FPWD.

I've strengthened the language, so it now says the Director MUST consider evidence. @michael-n-cooper suggested this for horizontal review, but for consistency and simplicity I've made this change for both wide and horizontal review.

I've changed the requirement for transitioning onto the Rec Track, so that either self-review or solicited review MUST happen. As @r12a notes, self-review is preferable, but I also agree with his point that we can't let the absence of questionnaires block us from improving the status quo.

I've removed the requirement for soliciting horizontal review within 28 days of FPWD publication, and have instead made it a MUST to send a notification to (@r12a's proposed email) horizontal-review-announce@ instead.

I've kept the 60 day notice for horizontal review before requesting transition to CR. This is a reasonable time-frame, and one that would be observed out of courtesy to the HR groups if only WG were thinking about it. It just isn't on their radar though, despite guidance and recommendations being available in allsorts of places, so putting this in the Process is really the only way to make it happen.

I've removed the section on the HR IG. I've kept the content, so I can reuse what's useful to write some implementation guidance elsewhere.

Dropping the HR IG section means the traffic light system is no longer mentioned at all, and consequently the requirement for specifications to receive the green light is also gone (per @michael-n-cooper's suggestion). The question is whether we want to make some form of sign-off by the horizontal review groups a SHOULD/MUST for CR transition?

[aphillips]

@r12a mentioned this in the I18N Teleconference yesterday and I assigned myself an action item to add to this thread. Obviously, I agree with his previous comments and want to double-down. Ensuring that every transreq has requested a review and monitoring for new working drafts is an extra burden. It is error prone and difficult.

And then there is the timing. It is so bad that the boilerplate for any review requests I do receive presupposes the need to yell at the sender:

Thanks for your request for review. Please note that we only perform the internationalization (I18N) portion of a wide review. I have added your specification to our review radar [1] and set the due date to {date}. Please advise if this date is acceptable. Although we will endeavor to meet your requested schedule, I have to note that 2-3 weeks is not enough time for a review, particularly if we make comments that require discussion. Please allow more time (we recommend starting this process at the FPWD stage) in the future. You may also wish to perform a self-review [2].

This is not a good customer experience. We need this to be in the Process because, as r12a calls out, nothing else has worked.

We need a clear "minimum window" of notification, because every request is an emergency. A disproportionate number of comments from I18N come from the chair or the staff contact because there isn't enough time to get a member of the WG to volunteer and then review the comments as a WG and then submit them. We have submitted substandard comments or incomplete reviews as a result. The receivers are under a lot more pressure to quash our comments, since they come at the last possible moment and are super inconvenient. It's not healthy.

I endorse @LJWatson's last comment that this would "be observed out of courtesy to the HR groups if only WG were thinking about it". We need to make it something that is tripped over. No one I'm aware of has gone from charter to CR in 60 days. This isn't a super-formulaic set of additional gates and waiting periods. We do not require or expect a frozen document or a pseudo-last call.

@LJWatson while I like WGs doing self-reviews, the current text makes it look as if a self-review might be sufficient for CR. It needs to be extremely clear that it is not. I prefer that a self-review be reviewed :-). So I would change this to be something like:

Before a proposed specification transitions onto the Recommendation Track, the group responsible for the specification MUST solicit a horizontal review from each of the horizontal review areas (in many cases, submitting a self-review to the reviewing group satisfies this requirement).

[LJWatson]

@aphillips commented:

while I like WGs doing self-reviews, the current text makes it look as if a self-review might be sufficient for CR. It needs to be extremely clear that it is not. I prefer that a self-review be reviewed :-). So I would change this to be something like: "Before a proposed specification transitions onto the Recommendation Track, the group responsible for the specification MUST solicit a horizontal review from each of the horizontal review areas (in many cases, submitting a self-review to the reviewing group satisfies this requirement).">

Good point. I've used your words to update the draft.

[astearns]

Should the 6.2.3.1 SHOULD list have a bullet point for ”Other standards organizations working in a relevant area”? The CSSWG will sometimes ask for input from Unicode Consortium or WHATWG members (for example), and @nigelmegitt mentions other orgs as well above https://github.com/w3c/w3process/issues/130#issuecomment-368804964

I suppose these additional organizations could be listed in a group’s charter, and thus be covered by the third bullet point. But for my case my current charter does not mention these orgs.