define "independent"

[nigelmegitt]

I've recently had cause to discuss what the Process's use of the term "independent" means, as applying to implementation experience.

The term is not defined in the Process, and some folk might want to bend what I consider to be a common sense interpretation to meet their own objectives. (Or maybe they have the common sense and I'm the one bending the interpretation!) There's room for clarification in relation to the following aspects:

• the organisation that has created the implementation (#9 might be related)
• the individuals involved in creating the implementation
• shared dependencies amongst implementations

I'm sure that the Director has some meaning of "independent" in mind; it would be helpful at least to set out some guidance, even if not to lock it down to a tight definition.

[dwsinger]

Indeed, are two implementations that don't share code, were written by different engineers, but are from the same company, 'independent'? (We have this case in WebVTT). If the same consultant wrote different implementations for two different companies, are they 'independent'? And so on. Saying a little what the goal of 'independence' is would be good (presumably, that the spec. is clear enough that implementations by people who merely read the spec. and didn't talk much to each other nonetheless interoperate).

[jeffjaffe]

A feature of the process is to allow the Director to apply judgment rather than rigorously defining every term. The problem with the latter is that it sometimes locks you in to a definition - and then a new case arrives which "should" be viewed as independent, but is somewhat different in its details. Before we change this it would be useful to have some use cases where this has been a problem in the past.

[vfournier17]

@jeffjaffe This should be an objective determination, rather than being one person's subjective decision. Members may not be comfortable with the Director (or any other person) making an individual determination regarding IP rights that will affect all members. I agree that it would help to look at use cases to help define the parameters of an independent implementation.

[wseltzer]

The term is used in a series of non-exhaustive examples and as a description of the goal, to show that a specification is sufficiently clear, complete, and relevant to market needs, to ensure that independent interoperable implementations of each feature of the specification will be realized."

On which of those do you want more clarity?

[nigelmegitt]

Hi @wseltzer I was thinking about the independence of implementations mainly. If the definition for that differs from the definition of other usages of the term, then that's another good reason for adding further clarification.

[wseltzer]

Has this been a problem in practice? if you can share a pointer to that discussion, it might help to indicate what clarity would help in the future.

[nigelmegitt]

@wseltzer Yes, it has. It was recently discussed in the TTWG and has previously been discussed there (private discussion, no link) in the context of CR exit criteria and implementation reports.

[wseltzer]

@plehegar or @swickr any input here?

[chaals]

@dwsinger wrote:

Saying a little what the goal of 'independence' is would be good (presumably, that the spec. is clear enough that implementations by people who merely read the spec. and didn't talk much to each other nonetheless interoperate)

That is exactly the goal, and why "independent" is hard to define here. It is probably not even a useful term, but it came with the old language that was even less about relevance to reality and more about being able to check some boxes and say "we met the criteria".

[michaelchampion]

I think I've made this point before, maybe in another thread. This language is a legacy of the "waterfall" era when the accepted work mode was to collaborate on an authoritative spec, then independently develop proprietary implementations. If the spec wasn't clear enough for at least 2 independent implementations of a feature to interoperate, that is evidence that the spec is not good enough.

That's still a useful criterion in situations where there are independent implementations, but we can't let it get in the way of the more common collaboration mode these days to collaborate on code and then document and standardize the "interoperability surface" APIs/protocols/formats. Even when there are multiple open source implementations such as the browser engines, they tend to look at each others code so demanding "independent" implementation experience would inhibit standardization.

So I agree with Chaals; it's probably not a useful term in the process document anymore. Or it is a useful abstract goal, but it's not a useful constraint on proving standards readiness. We've moved from a philosophy of "tests prove whether the spec is clear enough to implement" to "tests prove whether the spec is a good description of what interoperates across implementations".

[wseltzer]

This is a case-by-case determination for the Director, who is willing to discuss it with WG participants as needed, but we don't think adding more to the Process will be helpful.

[frivoal]

@nigelmegitt I am preparing a Disposition of Comments for this year's iteration on the Process. It was decided not to address this in Process 2019. Since you raised the initial issue, I would appreciate if you can let us know if this is something you can live with, or if you wish to object to.

[nigelmegitt]

Hi @frivoal clearly it is possible to live with this, since W3C has been doing so for some time. So an objection seems too strong and I won't object. I would like to register that the issue has not been resolved however.

Indeed, given the recent announcement by Microsoft about Edge, the question of independence of implementation seems more relevant than ever. Perhaps this is a wider topic that should be raised with the AC or the AB: what is important to W3C here?

[frivoal]

Thanks for the clarification. You make a good point, and I have raised an issue in the private AB tracker to think through what the decreasing diversity in browser engines meant for us.

[nigelmegitt]

For the record, the labels "Closed: rejected" and "Commenter satisfied" do a very bad job at reflecting the status of this issue and of my response. I am not satisfied, I just don't want to object to Process 2019 on that basis.

I would prefer that we consider it to be deferred pending further discussion, i.e. moved away from Process 2019, to reflect the purpose noted in https://github.com/w3c/w3process/issues/167#issuecomment-445447564.

[frivoal]

"Commenter satisfied" do a very bad job at reflecting the status of this issue and of my response

Point taken.

Separately from whether this should be deferred rather than rejected, commenter satisfied is meant to cover "proposal rejected, commenter confirms no objection". Do you have a suggestion of a better wording for the label itself?

---

I would prefer that we consider it to be deferred pending further discussion

I was just labeling issues after the fact, and this was just closed. I wasn't following this when the decision to close was taken (and I am not chair). @dwsinger with your chair hat on, is this deferred or rejected?

[dwsinger]

I believe that Wendy maintained, and the process CG agreed, that defining 'independent' clearly in the process would be hard, and take too much text. But indeed somewhere we need explanation or guidance.

[nigelmegitt]

Separately from whether this should be deferred rather than rejected, commenter satisfied is meant to cover "proposal rejected, commenter confirms no objection". Do you have a suggestion of a better wording for the label itself?

I think the problem is with the Closed: rejected label, when the response I gave was specifically in relation to the text "decided not to address this in Process 2019". I took that to mean a deferral rather than an outright rejection. In that scenario an approach I've seen before is to assign issues to a milestone, e.g. Process 2019 or Process 2020 or Process future and filter the issues for reporting based on the milestone. Of course in this case the issue was raised against Process 2019 so it does make sense to include it in the dispositions. Maybe a Deferred label could be helpful here.

Or the issue could be closed with a reference to a new issue specifically opened against a future process milestone, in which case closing this issue would be acceptable because the problem itself is still being tracked and worked on. I don't really like tracing through histories that span issues though.

[dwsinger]

I'm fine with leaving this open until such time as we understand how/where this is documented and defined. re-opening.

[dwsinger]

We should look at improving /Guide, probably in https://www.w3.org/Guide/standards-track/, to give guidance without setting rules. It's too situational for rules, but advice on such questions as:

• are two implementations, on different code bases, from the same company, 'independent'? (e.g. Apple has Webkit and CoreMedia implementations of VTT, written by different engineers)
• if everyone implements using an open-source library or polyfill, does that count (e.g. OpenSSL, though think of Heartbleed; and if it's a polyfill, that suggests we should have worked on the API surface that the polyfill rests on, not the format it interprets)?
• how 'independent' are products sharing an engine? if that engine forked from another engine in the past? (This addresses the concern that the new specification shouldn't favor a particular architecture, if possible).
• if two or more organizations sub-contracted the work to the same contractor, and they wrote afresh for each client, is that 'independent'?
• … and probably more

would probably be good.

[nigelmegitt]

@nrooney was leading work in the AB on rewriting /Guide so this would be useful to take on in the context of that work. Since the AB elections I'm not sure who is looking after that now.

I would add that some kind of scoping of "how much of the implementation has to be independent" would be useful. For example, if NewSpec specifies something whose implementation might itself be based on implementations of MatureSpec, then arguably only those parts that are new need to be independently implemented, so two implementations A and B of NewSpec that each happen to be dependent on the same (known good) implementation M of MatureSpec can still be independent, if written by different teams, with different funders etc.

But if A and B are based on the same implementation X of an AdHocPractice that isn't even close to being a Rec, then arguably they should be considered less independent.

[nigelmegitt]

New at this

@datbossg31030 Please could you elaborate a little more? I don't understand this comment as it is.

[dbaron]

So there are a few characteristics of standards that I think are valuable that I think the independence of the implementations helps verify. But those different characteristics depend on different definitions:

1. It's important to know that the specification (perhaps supplemented by the test suite) contains enough information to produce interoperable implementations without reverse-engineering an existing implementation. To verify this, it's important that the implementations be done by different people or by non-overlapping teams (since one person is likely to make the same assumptions across multiple implementations).

2. For specifications that are specifying a feature that is to be added to existing pieces of software, it's important to ensure that the specification doesn't make assumptions that are specific to one of those pieces of software that would make it much harder to add to a different one. To verify this, it's important that the implementations are done in different pieces of existing software. (This actually also helps to verify the previous criterion, since assumptions are also built into existing pieces of software.)

3. For some specifications, it's important to know that it has broad enough support within the industry that multiple companies are willing to invest in its success. One thing that can help verify this is if multiple companies are willing to invest in implementing it.

4. For some specifications, the existence of multiple implementations of a specification gives the users of that specification more confidence that they would be able to switch to a different implementation if they needed to (for example, to avoid a security vulnerability in one implementation). To verify this, it may be important not only that the implementations be in different pieces of software but also that they lack common dependencies.

I'm sure there are also more that I haven't listed here.

We seem to have settled on two independent implementations as the threshold for advancement to Proposed Recommendation. It's a clear threshold, but at the same time, for many of these things, three is better than two, etc. But if needed, more can also come after a specification advances to Proposed Recommendation.

[frivoal]

@dbaron I agree with your list, and that for many of these criteria, 3 or more is better than two.

The harder (to me) question is what to do when something has many deployments and broad adoption but only one implementation, such as the case where a single open source library is used by everybody to implement a particular feature.

This gives evidence to your point 3: even if they didn't implement it independently, multiple organizations found it worth shipping, and multiple organizations may have been willing to spend resources on developing / bugfixing the shared implementation.

The fact that it is adopted in multiple products also gives some evidence for point 2: if the spec and library could be used in multiple products, then that gives some assurance about too strong assumptions about a particular product architecture.

For point 1, it's fuzzy: while it's a single open source library, its various contributors, who may not be strongly coordinated, have settled on one interpretation of the spec. It may be by accident, the spec may still be ambiguous, etc, but it's better than nothing. In particular, if there are tests and the test have either been authored or reviewed by someone other than the implementer, this could have given the chance to notice any discrepancy between the spec and the implementation. It's not as strong as multiple implementations, but it's better than what a single closed-source implementation without a shared test suite would be.

For point 4, it's pretty bad: if that particular implementation turns out to have some bad security flaw, there's nothing you can switch to, without starting a new implementation at that point. That ought to be doable thanks to point 1 through 3, but it's not as good as knowing something already is available and could be adopted.

To, what this means is that for a standalone product implementing on its own a spec, we absolutely need more than one implementation to be able to graduate to PR and beyond.

But for when a spec corresponds to a feature inside a product rather than to the entirety of the product, multiple independent implementations are still desirable, but multiple integrations of a single implementation (or of several implementations which aren't fully independent) may be tolerable, even if it is not as good.

[plehegar]

will work on a guidlines document as an intermediate document

[hober]

See also https://github.com/w3c/oss-relations/issues/2 and https://github.com/w3c/AB-memberonly/issues/58

[hober]

See also #522

[nigelmegitt]

In particular: https://github.com/w3c/w3process/issues/522#issuecomment-1049628718

[nigelmegitt]

Breakout session at TPAC 2022: https://www.w3.org/2022/09/14-iii-minutes.html

[michaelchampion]

Thanks for the link to the breakout session. There seems to have been at least 3 different views expressed about what it should take to get to Recommendation:

1. The spec document itself is high quality, and COULD be interoperably implemented
2. The spec documents a technology or set of practices that SHOULD be implemented and deployed
3. The spec documents a technology that REALLY IS interoperably implemented and deployed in the real world

Is that a reasonable summary?
IMHO the answer should be "all 3". I'd eventually like to see success criteria at least as strong as CSS's https://www.w3.org/2020/12/css-wg-charter.html#success-criteria baked into the Process. I assume there is not broad consensus in the CG to put that in the Process yet 😢 . It will take some effort by the AB and TAG to make authoritative statements on the values a Recommendation SHOULD promote, and it will take some effort to define what "implemented and deployed in the real world" means, especially for guidelines documents that are normative rather than empirical.

[plehegar]

from https://www.w3.org/2023/01/11-w3process-minutes.html -> deferred

[nigelmegitt]

Is that a reasonable summary?

I don't think so, no, specifically because of the conjunction of "implemented" and "deployed in the real world" at step 3. Deployment should be broken out into a separate bullet, since it is independent from "implementation". There are many examples where standards work precedes deployment, and this is the reality of the commercial world: folk don't want to commit to something that it not yet standardised (whatever that means to them), even if they would support that standardisation activity and agree that the result would be beneficial.