Determine what works across browsers

[atopal]

What are the features that work across all major browsers? Do we have the data to say what the baseline of the web platform is? Is that the MDN compat data, WPT? Can we use the data to categorize features? If not, what do we need to do to get that data?

[foolip]

https://github.com/mdn/browser-compat-data would be a natural starting point for this. Using https://github.com/mdn/browser-compat-data/pull/13470 I've created a BCD "dashboard" with some filtering for Chrome/Firefox/Safari omissions.

Additional filtering would be pretty easy to add. This is a simplified view of the data, but if one added in the date that the feature was shipped in each browser, we could filter to features that have been available for N months in whichever browsers we like.

In my experience so far, looking closely at BCD always reveals mistakes in the data, and I think a manual review of the results would be critical.

[dontcallmedom]

does the checked state includes implementations marked as partial? behind flags or other restrictions?

could you provide a view that shows features available without restrictions in the last 2 main releases of the Chrome/Firefox/Safari, sorted from most recently available to least? This would be helpful in exploring the dynamics of a possible baseline.

[captainbrosset]

I love where this is going. Great dashboard @foolip! I'm a little concerned with the level of details available to us on BCD with regards to how much a feature is implemented in a browser. Do we have a sense for how much efforts would be involved in making sure the "partial" data is up to date before using the data?

For example, the css.properties.appearance.compat-auto is partially implemented in Safari, but there is no information as to how much of it is available. Maybe for narrow use cases like this it's fine to just assume it's not part of the baseline, but there may be much broader features that can be used to, maybe 50%, in some browsers while still being worth being part of the baseline.

The other thing I have in mind here is the browsers part of BCD is, for now, manually updated. I feel like we would need to make sure it is accurate and consistently updated with new browser builds. If we want to do any kind of filtering by number of months, then we need the dates to be correct. I started giving this some thoughts for Edge to start with, but I'm hoping that we can find a way to generalize this all browsers.

[foolip]

@dontcallmedom

does the checked state includes implementations marked as partial? behind flags or other restrictions?

I'm "flattening" support into a boolean value in https://github.com/mdn/browser-compat-data/blob/433b9ed85f3d58be1ef35a2562f3dcb2c127c967/csv.js#L57-L79

I'm filtering out anything behind a flag, but don't do anything about partial_implementation or notes, which will be misleading in cases where the notes say "this actually doesn't work at all" or similar.

could you provide a view that shows features available without restrictions in the last 2 main releases of the Chrome/Firefox/Safari, sorted from most recently available to least? This would be helpful in exploring the dynamics of a possible baseline.

By "without restrictions", do you mean filtering out anything marked as partial_implementation? That's easy, I can do that. And do you mean 2 main releases or the last 2 years of releases? If you do mean releases, what would you consider main releases for Safari? (That has recently gotten more complicated with more frequent releases, but not all of them update WebKit.)

[foolip]

@captainbrosset

I love where this is going. Great dashboard @foolip! I'm a little concerned with the level of details available to us on BCD with regards to how much a feature is implemented in a browser. Do we have a sense for how much efforts would be involved in making sure the "partial" data is up to date before using the data?

I'm glad you like it, it's a start I guess :) My hunch after having worked with BCD a lot is that when partial_implementation is used it's appropriate, but when it's missing that's not much of a guarantee the feature is solid. Links to bugs and use of the partial_implementation status happens mostly in response to issues filed. If you want to judge this for yourself, the experiment I'd suggest it sampling BCD features at random that have or don't have partial_implementation, and seeing if you agree with the data. Hopefully it's >90% correct, but I doubt it's >99% correct.

For example, the css.properties.appearance.compat-auto is partially implemented in Safari, but there is no information as to how much of it is available. Maybe for narrow use cases like this it's fine to just assume it's not part of the baseline, but there may be much broader features that can be used to, maybe 50%, in some browsers while still being worth being part of the baseline.

This specific piece of data is fixed in https://github.com/mdn/browser-compat-data/pull/16288 (timing is coincidence) and https://github.com/mdn/browser-compat-data/issues/4162 is about always requiring a note for partial_implementation. It would be nice to not have to interpret notes to determine what's baseline, but we should probably do that initially to make sure the data is good enough.

The other thing I have in mind here is the browsers part of BCD is, for now, manually updated. I feel like we would need to make sure it is accurate and consistently updated with new browser builds. If we want to do any kind of filtering by number of months, then we need the dates to be correct. I started giving this some thoughts for Edge to start with, but I'm hoping that we can find a way to generalize this all browsers.

https://github.com/mdn/browser-compat-data/issues/1664 is about this, but I actually just closed it because it didn't know what to actually do about it. But any work to automate creating PRs for browser releases would be very welcome. I can reopen the issue if you'd like, or you could just send PRs and ask for review.

[dontcallmedom]

By "without restrictions", do you mean filtering out anything marked as partial_implementation? That's easy, I can do that.

I meant primarily behind flags; I think the partial_implementation flags and their associated comments would be useful to surface in the dashboard, but filtering them out might be premature.

And do you mean 2 main releases or the last 2 years of releases? If you do mean releases, what would you consider main releases for Safari? (That has recently gotten more complicated with more frequent releases, but not all of them update WebKit.)

I guess maybe what I need is a way to play with various parameters of browser releases :)

[foolip]

@PaulKinlan you showed me a tool today that already does some of the things we're discussing here. Is that something you could make a public copy of?

[foolip]

@dontcallmedom how about https://docs.google.com/spreadsheets/d/1Yq3ckvWJxmuJAXdJmYlqGWAck06jBrKe3ZP4JNDhlhs/edit?resourcekey=0-d3ZxSzdCkO3mjmtYSHdggA#gid=1568658200&fvid=1085514526?

I haven't changed anything about partial_implementation, but I've stopped flattening to support to a boolean and instead use a version number, and also a date.

[atopal]

It looks like we need to break this down further. Maybe it would be helpful to go ahead and run a manual analysis with a random BCD sample from all areas? That would give us some data to then determine what we'd need to improve to be able to rely on the data set at scale, even if we have to allow for manual review.

[foolip]

I have an updated spreadsheet where it's possible to filter by dates and the number of browsers supporting each feature in BCD: https://docs.google.com/spreadsheets/d/1Yq3ckvWJxmuJAXdJmYlqGWAck06jBrKe3ZP4JNDhlhs/edit?resourcekey=0-d3ZxSzdCkO3mjmtYSHdggA#gid=1192153638&fvid=136189203

(I still haven't done anything to filter out subfeatures with identical data as requested by @dontcallmedom.)

[tidoust]

I'm working on w3c/browser-statuses, which seems relevant to this discussion. @dontcallmedom thought it might be useful to report on experience.

Context for the project is that individual and groups in W3C sometimes need to create overview and roadmap documents in their respective area to list features widely available, under development, being considered or identified as known gaps. Experience with these documents is that they take time to author, are very useful at first, but then maintenance is a pain point and they quickly become outdated. To facilitate authoring and maintenance of such documents, we developed a framework for web technology roadmaps a few years ago. This was used to create the Roadmap of web applications on mobile or the Overview of media technologies of the web for example (both of them are somewhat outdated as I write this).

The framework includes a mechanism to fetch and merge implementation status reports from main platform status projects (Can I Use, Chrome Platform Status, Browser Compat Data, and Webkit Status). Overview/Roadmap documents are often interested in coarse-grained implementation info about either an entire spec or a feature within a spec. The framework tries to compute that by looking at specific entries in platform status projects. See for instance the table in https://w3c.github.io/web-roadmaps/media/rendering.html#well-deployed-technologies

Mapping between spec and features and these projects is entirely manual in the w3c/web-roadmaps framework.

Early 2021, @dontcallmedom suggested to split out the feature implementation status part from the framework into its own project, and to look into automating the mapping. I started doing that, then a butterfly must have flown somewhere on Earth and I only got back to it a couple of weeks ago. The result is w3c/browser-statuses

The implementation data is for specs defined in w3c/browser-specs. For each spec, the code can now automatically propose mappings to platform status projects (based on the URL of the spec). This works well for BCD since BCD also now leverages browser-specs and thus uses the same URLs. Mapping to other platform status projects is not always as smooth. For instance, entries in Chrome Platform status are no longer updated once a feature has shipped and very often reference specs through old draft URLs (or initial readme documents or issues that introduced the feature).

Having mappings is not enough to compute a coarse-grained implementation status for an entire spec or a feature though. The entries may go too deep into technical details (typically the case with BCD and there's always some bit in an API that a browser does not support). Conversely, entries may not cover the entire spec or feature under consideration. In browser-statuses, some of the entries can be flagged with a representative flag, so that the system knows which entries it needs to take into account, and which entries it can basically ignore. I cannot think of a way to automate that identification process though. I did it for the few specs that came from the web-roadmaps project, but most mappings don't have that flag yet and browser-statuses reports the implementation status with a guess flag in most cases as a result to indicate that it may be missing something.

Similarly, while it's possible to define "features" in specs in browser-statuses, that process is 100% manual for now, including finding mappings for them.

w3c/browser-statuses is at early stage for now and not integrated anywhere, although I'm planning to switch to it in the roadmaps framework. If there are convergence points with needs that you may have identified, I would be happy to help with them. My experience is that it is difficult to find the right granularity for implementation statuses because there is never a "right" granularity level in practice, it always depends on the context in which the implementation status is going to be used.

On top of providing implementation status for specs and features, w3c/browser-statuses could be used to analyze possible discrepancies between platform status projects on the implementation status of a given spec or feature. I haven't looked into that recently. In case of discrepancy, the system considers that projects are more authoritative than others on certain user agents (e.g. Chrome Platform Status is authoritative for Chrome, BCD for Firefox, Webkit for Safari).

[foolip]

Thanks @tidoust, that looks awesome, and is relevant both to this issue and https://github.com/web-platform-dx/feature-set/issues/2. The manual mapping between "feature" and BCD entries is especially interesting to me. Do you think that things could be split up a bit to make this mapping reusable?

A while ago I filed https://github.com/w3c/webref/issues/597, so the organization I'm imagining is that one project is just about the features of the web platform, a tree of features with increasing increasing granularity. Typical cases might be "CSS > Flexbox > Flex gap" and "CSS > Grid > Subgrid".

Where to maintain the mapping between features and BCD I'm not sure, but keeping it in w3c/browser-statuses would be a fine start I think.

[voxpelli]

I have been looking at correlating BCD data and polyfill.io data to try and gain insight into which features that works across browsers thanks to being polyfillable.

In that work I also found some possible improvements of BCD data which I discussed with @foolip and which he raised some PR:s for.

I added the in process code in https://github.com/web-platform-dx/feature-set/pull/3 so that others can pull it down and experiment with it as well and I will get back to it later and have it output some more actionable conclusions, similar to what @foolip has done for the BCD data itself.

[foolip]

@tidoust I took a look at https://raw.githubusercontent.com/w3c/browser-statuses/main/index.json. Are the feature names in there chosen from some existing list of features, does it largely correspond to https://github.com/Fyrd/caniuse/tree/main/features-json?

[tidoust]

The list really grew organically as more roadmap documents were produced. Essentially, it contains all features that roadmap documents talk about. If the entries seem well-aligned with Can I Use, that wasn't done on purpose but that would confirm that the features in Can I Use are at the right granularity to "talk about web stuff" in documents targeted at developers.

[romainmenke]

What are the features that work across all major browsers? Do we have the data to say what the baseline of the web platform is? Is that the MDN compat data, WPT? Can we use the data to categorize features? If not, what do we need to do to get that data?

I have been looking at correlating BCD data and polyfill.io data to try and gain insight into which features that works across browsers thanks to being polyfillable.

see : https://github.com/mrhenry/web-tests#web-tests-

This has been my focus for the past ±3 years. (maintaining and adding to polyfill.io and postcss-preset-env) Happy to share my experiences with this.

[tidoust]

Closing this issue as "being addressed": this initial discussion sparked more concrete work in the web-features project and the adoption of a baseline status so that we can now talk about features that work across browsers. Still lots of work ahead, for sure ;)