search

web-platform-tests / interop

web-platform-tests Interop project
https://wpt.fyi/interop
278 stars 28 forks source link

Clarifying spec requirements for Interop 2025 #589

Open foolip opened 10 months ago

foolip commented 10 months ago

The Interop 2024 proposal template said this about the specification:

The feature must defined by a standards-track specification from one of the following organizations:

This issue tracks changes or refinements we want to make. So far:

foolip commented 10 months ago

For Wasm, I would suggest Phase 2+ as the bar.

jensimmons commented 10 months ago

Let's add:

In fact, I'd recommend adding them to the current process proposal. And adding an "etc" to the end. I don't believe we are intending to gate proposals by saying "oh that web standards org doesn't count, only these ones do". It seems our intention is to say "this must be standardized in an industry web standards org". And it's ok if we add some as we go, as we think of technology we want to include. The availability of tests will be more of a blocker than which standards org something is part of. If we someday want to exclude a particular standards org, then we can do so later.

How about changing the language to this:

The feature must defined by a standards-track specification from one of the following organizations:

jensimmons commented 10 months ago

I created a suggested amendment to the proposal PR at https://github.com/web-platform-tests/interop/pull/591

jgraham commented 10 months ago

We got the feedback internally that Phase 2 is too early for WASM.

For something to be in Interop we'd expect it to be reaching Phase 4 by the end of the year, since we'd presumably have multiple implementations all passing the testsuite. In practice that seems extremely unlikely unless something is in Phase 3 already, and accepting Phase 2 proposals risks rushing the standards process, and requiring difficult, incompatible, changes to already shipping features.

jensimmons commented 10 months ago

So is your recommendation that we say "WASM (Phase 3+)" or "WASM (Phase 4+" @jgraham ?

jgraham commented 10 months ago

Working backwards, I'd expect that after Interop a proposal would be in at least Phase 4 i.e. multiple implementations would pass the testsuite. Therefore that doesn't make sense as an precondition for being in Interop in the first place.

Phase 3 requires a testsuite to exist, which is a necessary precondition. It doesn't require complete formal spec text to exist, which does also seem like a requirement, and one we have for other specs in other groups.

So in practice I think "Phase 3" is the minimum bar, but for a proposal to be accepted, we'd additionally look for a usable specification, and agreement that it is unlikely to undergo substantive change. However since those additional criteria don't map cleanly to the phases, I think "Phase 3" is the right formal requirement.

jensimmons commented 10 months ago

I agree that any specification should be judged independently to assess whether it's actually ready or not. The levels are applied inconsistently, plus 'it depends' factors in heavily.

While we are at it... I don't think W3C requiring (Recommendation Track) is accurate. We often include things from W3C that are in Working Draft — if we deem they are ready. Rec is an incredibly high bar. And just not what we are doing.

No one is suggesting that if a standard has a certain level, then that's all it needs to be considered 'done enough'. We are defining a minimum bar, where any standard that doesn't have at least this should be considered disqualified.

zcorpan commented 10 months ago

@jensimmons Recommendation Track and Recommendation are different things, see https://www.w3.org/2023/Process-20230612/#rec-track

i.e. any of FPWD, WD, CR, PR, REC are rec-track.

jensimmons commented 10 months ago

Ah, you are correct. Nevermind my comment then. Yes, on track to become recommendation.

foolip commented 10 months ago

Let's go with Phase 3+ for WebAssembly then. I've suggested an edit at https://github.com/web-platform-tests/interop/pull/591#discussion_r1372852831

foolip commented 10 months ago

This has been integrated into https://github.com/web-platform-tests/interop/pull/591 now.

jsnkuhn commented 6 months ago

I'd like to recommend including links to explainers of what thing like "Recommendation Track" mean instead of links to the wider organizations involved. This explanation https://www.w3.org/2023/Process-20230612/#rec-track is considerably more useful than sending us generically to w3.org. Honestly, I've been in web dev for almost 20 years at this point and have only really started to grasp the process for specs since I started posting in the csswg github a few years ago. I'd imagine most devs would not be able to explain to you what "Recommendation Track" specifically means or even know where to go to get that explanation.