smoke-testing the values...

[chaals]

@darobin suggested in a comment on a different issue

... I do think that there's a good smoke test in the question "how would we have resolved the DRM issue using governing documents instead of Tim" and "are we satisfied that it's at least as good and ideally better".

I believe the core point of the smoke test is to think about "would we have come up with a much worse decision if we were using the vision instead of Tim as our guidance?", and if the answer is "yes, probably" then we clearly have work to do before the vision is ready.

IMHO the test is only relevant for contentious decisions, and DRM as a principle, EME as a concrete decision required, is an outstandingly, but not the only, relevant example.

There are some tricky points in here. First, IMHO, is "what does 'much worse' mean"? I think the answer isn't very technical - it's more about whether getting to a decision would have been even more divisive. A subsequent question and whether it is possible do better than that without a Director.

We apply DRM to video but won't apply it to written works doesn't seem to me clearly defensible on purely technical grounds. As @darobin said, it's one of those wicked hard social problems where the answer is a bit like "whatever causes the least friction", in this case choosing among a set of options that all cause a lot of friction.

Another historical example to consider is "would we have got to a Royalty-Free Patent Policy"? That's harder, because it was even longer ago (the EME decision is already clouded by the fog of memories), and the contemporary environment was quite different.

Discussions around DIDs, or whether we should have made the MoU with WHATWG for HTML and DOM, are more recent questions we could smoke test the same way.

[michaelchampion]

The relative priority of various values is a challenge that smoke testing might clarify. For example, "puts users first" is fine if there is a conflict between the apparent needs of users and spec writers. But what about user needs that can't be implemented with existing tech? (For example, "supports truth over falsehood, people over profits, humanity over hate").

And what about different user needs? I'm reminded of the joke/meme/wise saying "you can have it good, fast, cheap ... pick any two. " It is clear from the current draft that users' safety is the top priority, do we need to have similar language making it clear, for example, that user privacy is more important than low cost?

Other value conflicts are apparent in the current DID situation: the Vision mentions both "interoperability" and "reduce centralization in Web architecture, minimizing single points of failure and single points of control". How does one demonstrate interoperability without having a "single point of failure" protocol/API that all parties implement?

I'm not sure what the way forward is. It would be useful to prioritize values to guide resolution of such value conflicts. But that will make getting a consensus Vision document even harder. Or is there at least some meta-prioritizing the Vision doc could explain, to guide those trying to decide which value has a higher priority in a given situation?.

[darobin]

@michaelchampion That is why I have been suggesting that we build from the capabilities approach. We are not the first to have to contend with a planetary-scale project and have to make difficult decisions, e.g. given a limited budget do you spend more on education or on famine relief? I think that having the humility to learn from those who have had to solve these issues could serve us well.

It doesn't solve everything, all real-world institutional rules are incomplete, but it's a good north star and it's survived a lot of critical appraisal.

[michaelchampion]

@darobin are you suggesting that when there are conflicting values at stake, prioritize the ones W3C can actually do something about? For example, it can't create a truly decentralized identity system, but it CAN ensure that ID systems interoperate on the basis of some standard. Or in the EME case, it couldn't create a system for compensating artists with DRM-free media, but it could ensure that DRM systems interoperate via EME ...

Smoke testing the Vision with such a decision rule probably would be enlightening. That's more or less what I was trying to say with

some meta-prioritizing the Vision doc could explain, to guide those trying to decide which value has a higher priority in a given situation?.

[darobin]

No, I'm suggesting that we adopt the capabilities approach, focusing on substantive freedoms (which is very aligned with user agency). For DIDs, you'd want to run a more thorough analysis of user needs but capture-resistance and voice over the aspects that matter to them is more important to user than technical decentralisation. For EME, the question is what do users get in exchange for the sovereignty they abandon when they accept DRM? Cory is right that it leads to a device that is not entirely yours. That can be acceptable given appropriate rules and control by the affected parties. The right to use DRM should be dependent on signing a contract that guarantees a number of key rights, subject to regular renegociation.

[TzviyaSiegman]

Great discussion. Weighing relative values is important, but how do we capture this (briefly) in the Vision? Should we explicitly state (as in EWP) that "there are benefits and tradeoffs that may need to be carefully balanced."?

[chaals]

If we aim to use this as an operative guide to how we make these trade-off decisions, then there needs to be a lot more concrete guidance in how to make trade-off decisions than just a brief statement noting it's difficult.

Alternately, we could be clear that we rely on various elected and unelected groups to make the decisions - we're merely presenting the things that we expect them to be considering important, and we will defer to their judgement for any actual decision on a concrete question.

A drawback with the latter approach is that it makes the decision-making procedures much more contentious. I would expect it to become more common that people resort to lawyers and ambit claims in the resolution of formal objections, and that doesn't sound like a very positive direction.

[michaelchampion]

I don’t know how to put this in Vision language but maybe:

• say more about the priorities of values when there is rough consensus (e.g users first, security above convenience, accessibility a MUST, long established principles are generally higher priority than newly established ones…
• when there is a values conflict, decisions on which to prioritize must be EXPLAINED clearly and respectfully
• bottom line is to choose which alternative has the most evidence that choosing it will demonstrably improve the experience of a specific set of web users

[chaals]

the most evidence that choosing it will demonstrably improve the experience of a specific set of web users

Which group? How do we decide between e.g. "users who want better privacy above all" and "users who want better services above all" if that's the trade-off? does it depend on the size of th groups, or their guesstimated purchasing power, or where they are based?

The point of smoke-testing the values (which is what this issue is about) is that if we can't work out how they will answer these questions, it seems they are not yet usable as operational principles. If they don't obviously come up with something that we recognise as decisions at least as good as what we got in reality for various contentious issues, then we need to work out whether that's the best we can do.

So far, I think we'd struggle to even arrive at a decision, let alone a good one, without basing it mainly on extra context or assumptions and positions already held.

[chaals]

@michaelchampion suggested:

when there is a values conflict, decisions on which to prioritize must be EXPLAINED clearly and respectfully

I think this definitely needs to be part of how to manage apparent conflicts between values.

[chrisn]

If we aim to use this as an operative guide to how we make these trade-off decisions, then there needs to be a lot more concrete guidance in how to make trade-off decisions than just a brief statement noting it's difficult.

I agree. In https://github.com/w3c/AB-public/issues/96 I suggested reviewing the principles behind the EME decision. This decision involved a balancing of a number of different considerations and competing forces - which I don't know if we can capture succinctly in the set of high level principles we currently have in the Vision, but would be needed if we want the Vision to be more useful as guidance for deciding on contentious issues and formal objections.

Is this what we want from the Vision?

[AramZS]

I think doing a smoke test is a really good idea but, perhaps because my relative newness to the body working on this, I think it could use some additional set up here, some definition of what process we want to use for the smoke test and which situations we want to use from the past to process through iterations of this and potentially other documents. As someone less familiar with the previous decisions that have been dealt with I think it would be very useful.

[chaals]

A first thing is to pretend you are a member of the AB or TAG (whichever you prefer. They are really just people who get elected), look at the document, and work out if it helps you decide whether and why to uphold or overrule a formal objection, and how it guides you to an answer.

A second will be to do that as a group exercise, see what we can come up with as a decision, and then ask people to work out whether the decision and rationale are about as good as what we got. If so, this is a good sign. If not, we need to work out whether there are improvements we can make to the document that would help.

I think it makes sense to run this over a few decisions. In particular, ones where the proponents/objectors are not the same group of people, as a way to figure out how it works both when we are inclined to agree and to disagree with the formal objection or the original decision.

[mnot]

This issue seems to be focusing on testing the application of the values to decision-making, which is appropriate.

However, there are other ways we might use the values -- e.g., in determining how we allocate Team resources. That relates to how we measure the W3C's impact; see https://github.com/w3c/board/issues/110

[TzviyaSiegman]

This is an interesting discussion, but is there anything to add or change in the current draft @chaals @michaelchampion

[chaals]

If you think the current draft is good enough to guide decisions, let's ask people to pick it up and try some out, so we get some results of the proposed test. If not, then presumably there are things you want to change, and we should wait for that to happen and then take this up.

[dwsinger]

let's also see how well it fits into the pyramid – Mission statement, this document, and the more detailed documents on various fields (privacy, ethical web, and so on). That's also a test – are we consistent?

[michaelchampion]

I don't have suggestions for how to improve the current draft. I supported Chaals' question: do those of you on the Team, AB and TAG find the current vision draft useful when considering what work to start or advance?

Some current hard problems (and likely future ones) include:

• Solid and DID charters, which ran into a bunch of formal objections and controversy within the communities proposing them.
• What to do about the SVG WG, which doesn't appear to have critical mass of people doing the work although few dispute that SOMEBODY should work on it.
• Whatever the big controversy du jour is in eternal tension between a "free" web funded by advertising and a privacy-preserving web.

Long ago I hoped for a critical mass of the community that would support a Vision that at least laid a foundation to build consensus decisions for such hard problems on. Has the Vision helped guide thinking about concrete problems?

If not maybe "smoke testing" is too much to ask for, and the best the Vision can do is tell a consistent (as as @dwsinger puts it) if abstract story about W3C values, guiding principles, and a vision of how they could shape the web's future.

[dontcallmedom]

One idea for smoke testing the goal of the vision to "be opinionated enough to provide a framework for making decisions, particularly on controversial issues" would be to go through the list of existing group charters, and characterize if and how they help achieve W3C's vision for the Web.

Doing so might also help identify how some of these aspects could be surfaced operationally e.g. in the charter template, or in the innovation pipeline templates, or in the context of AC reviews of charters, etc.

It would be particularly interesting to identify current work that doesn't quite fit in that vision as this may either highlight ill-fitting work, or gaps in the vision.

[sideshowbarker]

there are other ways we might use the values -- e.g., in determining how we allocate Team resources. That relates to how we measure the W3C's impact; see w3c/board#110

💯 Strongly agree — and in my comment at https://github.com/w3c/board/issues/110#issuecomment-2044239322 I suggested one great way W3C could move forward on that is by focusing it into a list of 10 or so specific questions, and publishing those questions as an actual W3C document.

[tantek]

It has been a while since the most recent discussions on this issue.

Since then, I will note that I have heard anecdotal experience from the Team and others regarding using the W3C Vision Note in their decision-making and they have found it quite useful, in everything from chartering, to resolving objections (often amciably), to recruiting.

Thus I propose that we close this issue as complete or complete enough to proceed to Statement.

If we get new information or new experiences (where people tried to use the Vision to make hard decisions and found it lacking), then we should open new issue(s) for those specific opportunities for improvement.

(Originally published at: https://tantek.com/2024/269/t5/)

[cwilso]

As per Vision meeting at TPAC: we have smoke-tested the Vision values and believe they are solid enough to provide guidance. We may open new issues in the future.

[chaals]

It seems like a very thin basis for claiming this is done... Since I don't get to vote on making somethng a statement if it happens today, my opinion might be ignored. However, I think closing this issue on the basis of a bit of hearsay that "it's good" wouldn't actually match the vision we have of how we do things, and I request that this issue be reopened and considered a bit more carefully.

[darobin]

@chaals, I think you need to consider two points here:

1. As a stakeholder in our community, you may not have a vote but you do have a say as to whether you believe that the group has provided you with a substantive response including a sound rationale.
2. Evidence of wide review expected from a document should match our expectations as to how important the document is. For the two other documents we have on Statement Track — the EWP and the PP — we considered that a high bar needed to be met because these are impactful documents that we intend to enforce strictly as part of key processes such as horizontal review. In both cases we went far out of our way to solicit feedback and were much more thorough than you would be for, say, a typical browser feature Rec transition. But the AB may have very good reasons to believe that their vision document should meet a much lower bar? We shouldn't presume to know what they've decided between themselves.