search

web-platform-dx / web-features

Exploring how to present Web platform features adoptability
https://web-platform-dx.github.io/web-features/
Apache License 2.0
368 stars 72 forks source link

Baseline Survey #208

Closed dfabulich closed 1 month ago

dfabulich commented 1 year ago

There's been a lot of discussion around the definition of the Dynamic Baseline here on this repository, especially #174.

I'd like to use this issue to try to brainstorm ideas for a developer survey or interview approach, something that would allow us to use data to move forward on some of our open questions.

Background

Baseline is defined here:

https://web.dev/baseline/ https://web.dev/introducing-baseline/

To summarize, web developers have said in surveys that "keeping up with changes to the web platform or web standards" is their biggest pain point. https://web.dev/deep-dive-into-developer-pain-points/ (This data is from 2021, and I guess it could use a refresh, but I think there's broad agreement that this is still a major pain point in 2023.)

The goal of Baseline is to make it easier to keep up with changes.

The stated approach is to find a definition of Baseline that will "work for most developers most of the time." (I've criticized this guideline in #174, because it's not possible to use data to decide whether a feature does or doesn't work for a developer most of the time.)

With those goals in mind, Baseline was initially defined as "features that are supported in 2+ major versions." It's a clear, simple definition, supported by survey results.

The survey results that were used to support the "2+ major version" definition are available here: https://github.com/web-platform-dx/developer-research/blob/main/vendor-research/browser-support-matrix-survey.pdf (Be sure to download the PDF; Github does NOT render all pages directly on that page!)

Questions to investigate

Note that most of the questions below are about the variations on polyfill vs. substitute vs. progressive enhancement. It's possible that the survey could be a lot shorter if we want to elide these questions. (But I think it makes sense to keep these questions, because if you ask me "when would you use a new web feature?" the answer would have to be, "It depends. Does it have a good polyfill? A substitute? Can I progressively enhance it?")

A rough draft at proposed survey questions

This survey is about how and when you'd decide to start using a new browser feature, where not all users have upgraded to browsers that natively support the feature.

Question: Pollyfillable Feature

Imagine a new Web API feature that can be safely and correctly polyfilled, allowing the feature to work on older browsers that weren't designed to support the feature. The polyfill supports all users on all browsers, but not all users have browsers that natively support the feature.

How would you decide when to use a feature like this?

Question: Substitutable Feature

Imagine a new CSS feature. This feature can't be polyfilled, and not all users have browsers that support the feature natively, but you could manually substitute for the feature by applying CSS classes to your HTML instead.

How would you decide when to use this new CSS feature?

Question: Progressive Enhancement

Imagine a new Web API feature that can't be polyfilled, and only some of your users have a browser that supports the feature. But it is possible to use feature detection to progressively enhance your site, making the feature available to some users, but not all users.

(Let's suppose that the feature is not "essential" to the functionality of your web site or web app. The feature is valuable to users who can access it, but your site/app still does something useful for users who can't access the feature.)

How would you decide when to use a feature like this?

Question: Essential Features

Imagine a web feature that can't be polyfilled or transpiled, and only some of your users have a browser that supports the feature.

But imagine that it's not possible to progressively enhance this feature. If you use this feature, your site will be "broken" on any browser that doesn't support the feature.

How would you decide when to use a feature like this?

Question: Removing Polyfills

Imagine a web feature that you're using today with a polyfill. Today, all major browsers support the feature natively, and you're considering removing the polyfill, to rely on each browser's native built-in support.

But, imagine that removing the polyfill comes with a risk: if any user navigates to your site on an older browser that doesn't support the feature, your site/app will be broken.

How would you decide when to remove this polyfill?

Question: Market share data

If you use market share data to keep track of browsers and/or browser versions, from where do you get your data? (You may check more than one.)

[ ] I don't use browser market share data [ ] My own web logs [ ] Google Analytics [ ] StatCounter (gs.statcounter.com) [ ] Caniuse.com [ ] Browserslist [ ] Other _____

foolip commented 1 year ago

I'm not a researcher, but one thing I've heard from @atopal (also not a researcher) is that asking people to predict what they would do is a poor guide for what they will do, and that asking what they have done works better. This seems very plausible.

For example, questions on the form "In the past 12 months, how many times have you encountered situation X?" or "When you last encountered situation X, what did you do?"

I think most of the question can be rephrased to be retrospective rather than forward-looking, so this should be an easy fix.

foolip commented 1 year ago

My other feedback is that I think most developers simply don't know how many releases they need to support, how much time needs to pass, or what availability threshold they need. The most common process is probably testing it in one or a few browsers manually, shipping it if it works, and reacting to the loudest complaints of breakage. Even when consulting MDN compat tables or caniuse.com, it's a question of eyeballing how green it is, not a principled approach.

That being said, even if we can't have high confidence in what developers actually do based on a survey, we can learn something about what they expect from a reasonable Baseline definition, which is also valuable.

tidoust commented 1 year ago

Discussed during the 27 July baseline meeting

ddbeck commented 1 month ago

This is one of several issues around developing the Baseline definition, from last year. I don't think there any active plans to run a survey along these lines.

Now that we have a shipping definition of Baseline, we might have opportunities to run surveys on how Baseline is actually being used and understood, but "would you…"-style questions are probably no longer a likely avenue for investigation.

If anyone is reading this issue in 2024 or later: continuing to learn about how developers use and understand is important. If you have specific questions to raise and answer about developer expectations with regards to Baseline, please file a new issue. Thank you!