Missing specs from BCD spec_urls

[Elchi3]

Over in https://github.com/mdn/yari/pull/3518#issuecomment-823325691 I'm working on a SpecificationTable renderer for MDN that uses BCD spec_urls and w3c/browser-specs to display relevant specifications on MDN reference pages.

Given a BCD spec url like https://drafts.csswg.org/css-box/#margin, I'm querying w3c/browser-specs for additional information:

find((w3cspec) => 
  bcdurl.startsWith(w3cspec.url) || 
  bcdurl.startsWith(w3cspec.nightly.url) ||
  bcdurl.startsWith(w3cspec.series.nightlyUrl)
);

That way, I can then render name of the spec or a link the spec's test repository to MDN, for example.

While doing this, I noticed I can't find specs or nightly specs for the following URLs that are in BCD:

Is it possible to add them or should the BCD spec_urls use different URLs?

[
      'https://wicg.github.io/controls-list/',
      'https://datatracker.ietf.org/doc/html/rfc2397',
      'https://datatracker.ietf.org/doc/html/rfc8942',
      'https://datatracker.ietf.org/doc/html/rfc7231',
      'https://datatracker.ietf.org/doc/html/rfc7233',
      'https://datatracker.ietf.org/doc/html/rfc7234',
      'https://datatracker.ietf.org/doc/html/rfc7838',
      'https://datatracker.ietf.org/doc/html/rfc8246',
      'https://datatracker.ietf.org/doc/html/rfc7230',
      'https://datatracker.ietf.org/doc/html/rfc6266',
      'https://datatracker.ietf.org/doc/html/rfc7578',
      'https://datatracker.ietf.org/doc/html/rfc6265',
      'https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-digest-headers-05',
      'https://datatracker.ietf.org/doc/html/rfc8470',
      'https://datatracker.ietf.org/doc/html/rfc7232',
      'https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-expect-ct-08',
      'https://datatracker.ietf.org/doc/html/rfc7239',
      'https://datatracker.ietf.org/doc/html/draft-thomson-hybi-http-timeout-03',
      'https://datatracker.ietf.org/doc/html/rfc6454',
      'https://datatracker.ietf.org/doc/html/rfc7235',
      'https://datatracker.ietf.org/doc/html/rfc7469',
      'https://datatracker.ietf.org/doc/html/rfc6797',
      'https://datatracker.ietf.org/doc/html/rfc7540',
      'https://datatracker.ietf.org/doc/html/rfc7034',
      'https://datatracker.ietf.org/doc/html/rfc7538',
      'https://datatracker.ietf.org/doc/html/rfc2324',
      'https://datatracker.ietf.org/doc/html/rfc7725',
      'https://github.com/tc39/proposal-regexp-legacy-features/',
      'https://webassembly.github.io/threads/js-api/',
      'https://tc39.es/proposal-hashbang/out.html',
      'https://mathml-refresh.github.io/mathml/',
]

[dontcallmedom]

The CSS, Houdini and FXTF specs are in browser-specs, but using a versioned shortname (incl for editors draft) - I think we'll want to look into making them findable without the version one way or another.

I expect that the TC39.es/proposal-* URLs (maybe except for the -intl- ones?) are all stage 4, which means they're already integrated in the main spec - in that case, our assumption has been that referring to the main spec was the better approach, but if this is a bad assumption, happy to discuss.

https://www.w3.org/TR/CSP2/ has a more recent version https://www.w3.org/TR/CSP3/

https://webassembly.org/docs/web/ should probably be https://webassembly.github.io/spec/web-api/ ?

For IETF specs, we hadn't added them so far since they felt at a different layer from what we've been dealing with so far, but I don't think it would be problematic to add them.

Re DNT, I guess the case can be made that DNT has had adoption in browsers, so should likely just be added.

Re XPath3, XSLT3, can you say which pages/BCD features need these specs? I'm surprised that XPath3 and XSLT3 were ever adopted in browsers.

cc @tidoust

[dontcallmedom]

I forgot https://github.com/WebAssembly/threads/blob/master/proposals/threads/Overview.md in my review - which BCD feature needs this?

[dontcallmedom]

the client hint URL should be replaced with https://datatracker.ietf.org/doc/html/rfc8942

[Elchi3]

The CSS, Houdini and FXTF specs are in browser-specs, but using a versioned shortname (incl for editors draft) - I think we'll want to look into making them findable without the version one way or another.

Yes! I think it would be great if browser-specs would make these findable.

I expect that the TC39.es/proposal-* URLs (maybe except for the -intl- ones?) are all stage 4, which means they're already integrated in the main spec - in that case, our assumption has been that referring to the main spec was the better approach, but if this is a bad assumption, happy to discuss.

You're right, all except the pipeline operator are available in the main specs. I've sent pull requests to BCD to correct this.

https://www.w3.org/TR/CSP2/ has a more recent version https://www.w3.org/TR/CSP3/

In this case it is about https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/plugin-types which is in CSP2 only afaict. I think we sometimes run into case where a new spec doesn't have deprecated features. Maybe it should be filed against CSP3 so that it talks about deprecation at least? Would that apply generally when this situation happens?

https://webassembly.org/docs/web/ should probably be https://webassembly.github.io/spec/web-api/ ?

Yep, thanks, sent a BCD PR.

For IETF specs, we hadn't added them so far since they felt at a different layer from what we've been dealing with so far, but I don't think it would be problematic to add them.

👍

Re DNT, I guess the case can be made that DNT has had adoption in browsers, so should likely just be added.

👍

Re XPath3, XSLT3, can you say which pages/BCD features need these specs? I'm surprised that XPath3 and XSLT3 were ever adopted in browsers.

Oh yeah, no need to worry about these actually. We have a PR ready to remove the stub data for XPath3 and XSLT3 from BCD.

the client hint URL should be replaced with https://datatracker.ietf.org/doc/html/rfc8942

This was about the Save-Data header, I sent a PR to link to https://wicg.github.io/savedata/#save-data-request-header-field

[Elchi3]

Thanks a lot for your review, Dom! @sideshowbarker and I are still adding spec urls to BCD, so the list of spec hosts isn't exhaustive yet but I have the feeling it should be manageable and browser-specs would work for serving MDN the spec meta data.

[dontcallmedom]

In this case it is about https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/plugin-types which is in CSP2 only afaict. I think we sometimes run into case where a new spec doesn't have deprecated features. Maybe it should be filed against CSP3 so that it talks about deprecation at least? Would that apply generally when this situation happens?

a good question - it feels like its deprecation should indeed be documented in CSP3. Would you mind raising an issue there? if this doesn't fly for some reason, we can look into how to handle that situation in browser-specs.

[dontcallmedom]

all except the pipeline operator are available in the main specs

so https://tc39.es/proposal-pipeline-operator/ is Stage 1; our current approach has been to add Stage 3 specs automatically to browser-specs; I don't think there would be an issue with slurping data from a Stage 1 (or at least from that one since it is formatted as a spec), but I'm surprised of the need - is it already deployed in browsers? if so, why is it "only" stage 1?

[Elchi3]

so https://tc39.es/proposal-pipeline-operator/ is Stage 1; our current approach has been to add Stage 3 specs automatically to browser-specs; I don't think there would be an issue with slurping data from a Stage 1 (or at least from that one since it is formatted as a spec), but I'm surprised of the need - is it already deployed in browsers? if so, why is it "only" stage 1?

Generally we advice against documenting stage 1 features but when MDN was a wiki still, some people's enthusiasm for certain features couldn't be stopped to document things early :) That said, we can probably go with your approach to only add Stage 3 specs and kindly ask authors to wait until stage 3 before documenting things.

[dontcallmedom]

I don't think we really want https://datatracker.ietf.org/doc/html/rfc2324 in - "Hyper Text Coffee Pot Control Protocol", published on April 1st 1998…

[Elchi3]

I guess it doesn't qualify for this dataset. In the docs, it helps people to understand the joke https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/418

[Elchi3]

Updated the list above given the new browser-specs release and the updated BCD entries.

https://webassembly.github.io/threads/js-api/index.html

Not sure if this qualifies for addition. It's for the shared flag in https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly/Memory/Memory.

https://wicg.github.io/controls-list/

This is for https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/controlsList and it isn't really a spec. I think here the content should be updated to say that it isn't a spec but a draft pull request.

https://www.w3.org/TR/MathML3/chapter3.html

This is for https://developer.mozilla.org/en-US/docs/Web/MathML/Element/menclose which apparently isn't in the MathML Core spec.

[dontcallmedom]

https://www.w3.org/TR/MathML3/chapter3.html

This is for https://developer.mozilla.org/en-US/docs/Web/MathML/Element/menclose which apparently isn't in the MathML Core spec.

We looked a bit in the <menclose> situation (which is indeed implemented in Firefox & Safari) - it is under consideration for MathML Core, but probably not in the first level (although I'm not sure what that exactly means).

Adding MathML3 in browser-specs would be possible, but probably somewhat confusing; my sense is that is that it might be better for BCD to use a fallback mechanism for that particular case (which presumably may apply in other cases as well) - maybe, with a link to the relevant MathML Core issue to check progress on the topic.

[tidoust]

Both https://drafts.csswg.org/css-fonts-5/ and https://drafts.csswg.org/web-animations-2/ could be added to browser-specs if needed. That said, they have not yet been added because they both feature a "No ready for implementation" banner that asserts that the specs are just there to "record ideas". The second one is also flagged as an "Unofficial Proposal Draft". It would be good to have these bits updated in the spec before they get added to browser-specs (and BCD?)

[sideshowbarker]

size-adjust is marked experimental in BCD, and I’ve raised https://github.com/mdn/content/pull/4689 to mark it experimental in MDN, along with raising https://github.com/mdn/browser-compat-data/pull/10270 to mark KeyframeEffect.p.iterationComposite experimental in BCD.

These cases seem like pretty much exactly what the experimental flag in BCD is for — and what the Experimental banner in MDN is for. And even if were to remove them from BCD, we’d still have the MDN articles for them — they just wouldn’t have anything in the Browser Compatibility section. So then next we’d need to consider whether to remove the MDN articles, which would seem to not be a great outcome.

The fact that a particular feature has been experimentally implemented seems like a useful and appropriate piece of data to have in BCD — and it seems like not a bad thing to have an MDN article that documents the feature for developers who want to try it out and test it and hopefully in some cases maybe even file issues in the spec issue tracker to describe problems they’ve run into with the design and maybe to propose spec changes so that it’s specced in a way that works well for them.

So while I’m sure we can special-case https://drafts.csswg.org/css-fonts-5/ and https://drafts.csswg.org/web-animations-2/ drafts in the MDN/Yari tooling even if they’re not in browser-specs, these seem like instances that suggest the browsers-specs inclusion criteria might need to be adjusted to accommodate cases where despite a spec having a “Not ready for implementation” banner, features from the spec have in fact been experimentally implemented and shipped (and added to BCD, and documented in MDN).

[dontcallmedom]

the fact that size-adjust has experimental implementations in FF feels like a good reason to add css-font-5 despite the "not ready for implementation" banner, and likewise for web-animations-2, so 👍from me

[tidoust]

The CSS, Houdini and FXTF specs are in browser-specs, but using a versioned shortname (incl for editors draft) - I think we'll want to look into making them findable without the version one way or another.

Yes! I think it would be great if browser-specs would make these findable.

Now done and released in v1.37.0. You may match on the versionless URL looking at series.nightlyUrl (and series.releaseUrl for /TR URLs but I suspect you'll be more interested by the nightly ones). By definition, all specs in a series will have the same series.releaseUrl. The "current" one is the entry for which shortname equals series.currentSpecification.

That should take care of all the CSS specs that appear in the list at the top of this issue.

[Elchi3]

Thanks so much, @tidoust 👍 🎉 This is really cool!

Yes, for our purposes, the series.nightlyUrl is the important one.

I've updated the first comment to show how we should find specs now and which remain missing in browser-specs for the moment: https://github.com/w3c/browser-specs/issues/279#issue-863702092 (really small list now, thanks again!)

[dontcallmedom]

with #280 closed, I believe this issue can be closed - additional gaps should be (and have already been) reported in new issues

[Elchi3]

Thanks for all your work @dontcallmedom and @tidoust 👍 🚀

We're tracking exceptions in this file: https://github.com/mdn/browser-compat-data/blob/main/test/spec-urls.test.js#L27 and I'll remove the IETF specs there soon then.

[dontcallmedom]

removing the exceptions will probably need a new package release (which hasn't happen yet fyi)