Ensure all specs & standards have published user research?

[edent]

I am concerned about the number of specifications which rely on private user research. Would it be possible to add something to the process that all research conducted on a proposal be published in the open?

Without wishing to put the spotlight on any specific proposals, it appears that several have either done no user research (in which case, I think TAG should not approve) or have research which is private (in which case, it should be published before TAG can review).

Without this step, W3C risks endorsing standards which are not suitable for user needs. Or, disproportionately favours the authors.

I appreciate that there may be a privacy element to this. I'm not proposing that video interviews with users be made public. But rather the typical summary that would accompany any project. For example: number of people interviewed, age ranges, responses, qualitative and quantitative feedback, etc.

This would be a simple and useful way to make sure that any new proposal meets users' needs, and that all implementers of a proposal are able to understand the research behind it.

Thanks

[LeaVerou]

As someone with a background in HCI research, I would personally love a requirement that all specs have published user research! Philosophically, all your arguments make total sense to me. I have often lamented how we (WGs) are designing (what is essentially text-based) interfaces in a vacuum, and wished we could do some rounds of user testing. I have been in far too many WG debates where people argue about what authors would find easier to use, with no authors actually trying their hands on these alternatives (often there are no authors even in the room!). These should not be theoretical debates!

However, with my WG member hat on, I do worry that a blanket requirement for user testing of everything would delay specifications tremendously, and authors already complain that standards move too slowly. Designing a proper lab study for a coding task takes time. I’m afraid it's not realistic to require this of every specification.

Also, do note that the TAG's role is advisory only. We cannot prevent browsers from implementing a specification, and we cannot really prevent a specification from moving forwards no matter our objections to it. If our requirements are seen as unrealistic, our review will just be requested less frequently (or be taken into account less seriously).

That said, it may be realistic to require that the relevant findings of any research that has influenced the specification be disclosed in the open. It may be more feasible to require this going forwards, as it may not be possible for user research that has already been conducted, depending on the consent form the subjects signed (though it is unlikely).

[edent]

Thanks Lea. I agree with you. I don't think authors should be compelled to run full lab-quality studies. But if they have done any research which supports the proposal, it should be made available.

And, yes, while it might be nice to get someone to do retrospective studies - this would only be a concern for proposals going forward.

[hober]

We talked about this on our plenary call today. We identified a couple of things the TAG can do, but we also think this is something that should be considered for the W3C Process itself. @LeaVerou will write a PR for our Explainer explainer, and I've written a PR for our design review issue templates https://github.com/w3ctag/design-reviews/pull/653. @torgo filed w3c/w3process#551 so the Process CG can take a look.

[codedfiles]

25

[LJWatson]

Without user research we have no way of telling if we're creating standards that benefit anyone except ourselves.

User research does not have to be in the form of moderated testing though. That's most often done once a product has reached a sufficient state of maturity to have something tangible to be tested - wireframes, designs, prototypes etc.

Informal or formal surveys are a low-fi form of user research that can be done at the requirements gathering stage. Even a simple question that asks users if they think the proposed feature will a) solve a problem they have, b) make things worse, or c) something else, provides indicative data that can be useful when shaping standards.

Focus groups could also be used, albeit a little unconventionally. Asking for lightning slots at relevant meetups, presenting the proposed feature, then asking for questions, comments, and feedback, would provide another useful dataset.

Desktop research is another low-fi option. Looking for existing data that supports the idea that users are requesting the proposed feature or describing a problem that the proposed feature will solve.

[torgo]

We've made various changes to our issue templates and to the explainer template which try to address this so we decided we can close this issue at this time.