search

WICG / ua-client-hints

Wouldn't it be nice if `User-Agent` was a (set of) client hints?
https://wicg.github.io/ua-client-hints/
Other
590 stars 77 forks source link
shipping-chromium

readme

User Agent Client Hints

This repository hosts the User Agent Client Hints (UA-CH) specification.

UA-CH introduces the following Client Hints HTTP headers and a corresponding JavaScript API:

Contributing

We welcome contributions in the form of new issues, comments on existing issues, and pull requests. Before getting started, please read our Contributing Guide and the Code of Conduct.

Explainer: Reducing User-Agent Granularity

A Problem

User agents identify themselves to servers as part of each HTTP request via the User-Agent header. This header's value has grown in both length and complexity over the years; a complicated dance between server-side sniffing to provide the right experience for the right devices on the one hand, and client-side spoofing in order to bypass incorrect or inconvenient sniffing on the other. Chrome on iOS, for instance, currently identifies itself as:

User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/69.0.3497.105 Mobile/15E148 Safari/605.1

While Chrome on Android sends something more like:

User-Agent: Mozilla/5.0 (Linux; Android 9; Pixel 2 XL Build/PPP3.180510.008) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36

There's a lot of entropy wrapped up in the UA string that is sent to servers by default, for all first- and third-party requests. This makes it an important part of fingerprinting schemes of all sorts, as servers can passively capture this information without the user agent’s, or more importantly the user’s, awareness or ability to intervene or prevent such collection.

Safari has recently taken some steps to reduce the entropy of the user agent string, initially locking it to a single value, period, and then backing off a bit due to developer feedback. This document proposes a mechanism which might allow user agents generally to be a little more aggressive.

Challenges

From a developer's standpoint, the detail available in the UA string is valuable, and they reasonably object to dropping it. The feedback to Safari's UA string freeze was right along these lines, noting four broad categories of use:

  1. Brand and version information (e.g. "Chrome 69") allows websites to work around known bugs in specific releases that aren't otherwise detectable. For example, implementations of Content Security Policy have varied wildly between vendors, and it's difficult to know what policy to send in an HTTP response without knowing what browser is responsible for its parsing and execution.

  2. Developers will often negotiate what content to send based on the user agent and platform. Some application frameworks, for instance, will style an application on iOS differently from the same application on Android in order to match each platform's aesthetic and design patterns (https://onsen.io/ was the first example in a quick skim of search results, but there are certainly others).

  3. Similarly to #1, OS revisions and architecture can be responsible for specific bugs which can be worked around in website's code, and narrowly useful for things like selecting appropriate executables for download (32 vs 64 bit, ARM vs Intel, etc). Model information is likewise useful when bugs are limited to particular kinds of devices.

  4. Sophisticated developers use model/make to tailor their sites to the capabilities of the device (e.g. Facebook Year Class) and to pinpoint performance bugs and regressions which sometimes are specific to model/make.

These are use cases that are interesting for us to support. Browsers certainly have bugs, and developers certainly would be well-served by being able to work around them. We'll keep these challenges in mind with the proposal that follows.

A Proposal

By default, servers should receive only the user agent's brand, significant version number, underlying platform name, and mobileness. Servers can opt-into receiving information about minor versions, the underlying operating system's major version, and details about the underlying architecture, bitness and device model. The user agent can make reasonable decisions about how to respond to sites' requests for more granular data. Browsers might even offer user-controlled settings to determine which hints or values should be allowed for some or all sites. We might accomplish this as follows:

  1. Browsers should deprecate the User-Agent string over time, initially locking bits of its value, and ramping up over time to lock the entire string to something generic for the device type. Chrome could perhaps send the following for mobile, regardless of the underlying device type:

    Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/71.0.0.0 Mobile Safari/537.36

    And the following for desktop, similarly irrespective of the underlying device characteristics such as bitness or architecture:

    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.0.0 Safari/537.36

    These strings would stay static, adding cruft to every request from now until forever.

    We can ratchet this deprecation over time, beginning by freezing the version numbers in the header, then removing platform and model information as developers migrate to the alternative mechanisms proposed below.

  2. Similarly, user agents would freeze the navigator.appVersion, navigator.platform, navigator.productSub, navigator.vendor, and navigator.userAgent attributes to appropriate values for the frozen User-Agent string.

  3. Browsers should introduce several new Client Hint header fields:

    1. The Sec-CH-UA header field represents the user agent's brand and significant version. For example:

      Sec-CH-UA: "Chrome"; v="73", "Chromium"; v="73", "?Not:Your Browser"; v="11"

      Note: See the GREASE-like discussion below for how we could anticipate the inevitable lies which user agents might want to tell in this field and to learn more about the admittedly odd looking "?Not:Your Browser"; v="11".

    2. The Sec-CH-UA-Arch header field represents the underlying architecture's instruction set. For example:

      Sec-CH-UA-Arch: "arm"

    3. The Sec-CH-UA-Bitness header field represents the underlying architecture's bitness (i.e., the size in bits of an integer or memory address). This could be used to determine which binary to serve for downloads.

      For example:

      Sec-CH-UA-Bitness: "64"

    4. The Sec-CH-UA-Form-Factors header field represents the form-factor of a device, historically represented as a <deviceCompat> token in the User-Agent string (e.g., "Tablet", "VR", etc.)

      For example:

      Sec-CH-UA-Form-Factors: "XR"

    5. The Sec-CH-UA-Mobile header field represents whether the user agent should receive a specifically "mobile" UX.

      Sec-CH-UA-Mobile: ?1

    6. The Sec-CH-UA-Model header field represents the user agent's underlying device model. For example:

      Sec-CH-UA-Model: "Pixel 2 XL"

    7. The Sec-CH-UA-Full-Version header field represents the user agent's full version.

      Sec-CH-UA-Full-Version: "73.1.2343B.TR"

      Advisement: Sec-CH-UA-Full-Version is deprecated and will be removed in the future. Developers should use Sec-CH-UA-Full-Version-List instead.

    8. The Sec-CH-UA-Platform header field represents the platform's brand and major version. For example:

      Sec-CH-UA-Platform: "Windows"

    9. The Sec-CH-UA-Full-Version-List header field represents the full version for each brand in its brand list. For example:

      Sec-CH-UA-Full-Version-List: "Microsoft Edge"; v="92.0.902.73", "Chromium"; v="92.0.4515.131", "?Not:Your Browser"; v="3.1.2.0"

  4. These client hints should also be exposed via JavaScript APIs via a new navigator.userAgentData attribute:

    dictionary NavigatorUABrandVersion {
  DOMString brand;    // "Google Chrome"
  DOMString version;  // "84"
};

dictionary UADataValues {
  FrozenArray<NavigatorUABrandVersion> brands; // [ {brand: "Google Chrome", version: "84"}, {brand: "Chromium", version: "84"} ]
  boolean mobile;             // true
  DOMString architecture;     // "arm"
  DOMString bitness;          // "64"
  DOMString formFactors;       // ["Automotive"]
  FrozenArray<NavigatorUABrandVersion> fullVersionList; // [ {brand: "Google Chrome", version: "84.0.4147.0"}, {brand: "Chromium", version: "84.0.4147"} ]
  DOMString model;            // "X644GTM"
  DOMString platform;         // "PhoneOS"
  DOMString platformVersion;  // "10A"
  DOMString uaFullVersion; // deprecated in favor of fullVersionList
};

[Exposed=(Window,Worker)]
interface NavigatorUAData {
  readonly attribute FrozenArray<NavigatorUABrandVersion> brands; // [ {brand: "Google Chrome", version: "84"}, {brand: "Chromium", version: "84"} ]
  readonly attribute boolean mobile; // false
  readonly attribute platform; // "PhoneOS"
  Promise<UADataValues> getHighEntropyValues(sequence<DOMString> hints); // { architecture: "arm", bitness: "64", model: "X644GTM", platform: "PhoneOS", platformVersion: "10A", formFactors: ["VR"], fullVersionList: [ {brand: "Google Chrome", version: "84.1.2.3"}, {brand: "Chromium", version: "84.1.2.3"}, {brand: "Not A;Brand", version: "101.3.2.9"} ] }
};

interface mixin NavigatorUA {
  [SecureContext] readonly attribute NavigatorUAData userAgentData;
};

Navigator includes NavigatorUA;
WorkerNavigator includes NavigatorUA;

    User agents can make intelligent decisions about what to reveal in each of these attributes. Top-level sites a user visits frequently (or installs!) might get more granular data than cross-origin, nested sites, for example. We could conceivably even inject a permission prompt between the site's request and the Promise's resolution, if we decided that was a reasonable approach.

User agents will attach the Sec-CH-UA header to every secure outgoing request by default, with a value that includes only the significant version (e.g. "Chrome"; v="69"). They will also attach Sec-CH-UA-Mobile and Sec-CH-UA-Platform headers by default. Servers can opt-into receiving more detailed UA information using the other available Client Hints, by delivering an Accept-CH header opt-in, that includes the information they are interested in.

Note the word "secure" in the paragraph above, and the SecureContext attribute in the IDL: these client hints will not be delivered to plaintext endpoints. Non-secure HTTP will receive only the reduced User-Agent string, now and forever.

For example...

A user agent's initial request to https://example.com will include the following request headers:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
            Chrome/71.0.0.0 Safari/537.36
Sec-CH-UA: "Chrome"; v="74", ";Not)Your=Browser"; v="13"
Sec-CH-UA-Mobile: ?0
Sec-CH-UA-Platform: “Windows”

If a server delivers the following response header:

Accept-CH: Sec-CH-UA-Full-Version-List, Sec-CH-UA-Arch

Then subsequent requests to https://example.com will include the following request headers:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
            Chrome/71.0.0.0 Safari/537.36
Sec-CH-UA: "Chrome"; v="74", ";Not)Your=Browser"; v="13"
Sec-CH-UA-Mobile: ?0
Sec-CH-UA-Platform: "Windows"
Sec-CH-UA-Full-Version-List: "Chrome"; v="74.0.3729.0", "Chromium"; v="74.0.3729.0", "?Not:Your Browser"; v="13.0.1.0"
Sec-CH-UA-Arch: "arm"

For use-cases where non-default hints are required on first request, the two Client Hints Reliability mechanisms provide a means to do so. For example, an example using Critical-CH follows.

The initial request:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
            Chrome/71.0.0.0 Safari/537.36
Sec-CH-UA: "Chrome"; v="74", ";Not)Your=Browser"; v="13"
Sec-CH-UA-Mobile: ?0
Sec-CH-UA-Platform: “Windows”

The server responds that the Sec-CH-UA-Full-Version-List is required on first-request in order to deliver some optimized resource, for example:

Accept-CH: Sec-CH-UA-Full-Version-List
Critical-CH: Sec-CH-UA-Full-Version-List

The client then retries the initial request with the requested hints:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
            Chrome/71.0.0.0 Safari/537.36
Sec-CH-UA: "Chrome"; v="74", ";Not)Your=Browser"; v="13"
Sec-CH-UA-Mobile: ?0
Sec-CH-UA-Platform: “Windows”
Sec-CH-UA-Full-Version-List: "Chrome"; v="74.0.3729.0", "Chromium"; v="74.0.3729.0", "?Not:Your Browser"; v="13.0.1.0"

The user agent can make reasonable decisions about when to honor requests for detailed user agent hints and first-parties can use Permissions-Policy to decide which third-parties they'd like to privilege with detailed user agent information. In any event, moving to this opt-in model means that the extent of a site's usage can be monitored and evaluated.

For developers that prefer using user agent information to make client-side decisions, they could use the JavaScript API:

  const uaData = navigator.userAgentData;
  const brands = uaData.brands;     // [ {brand: "Google Chrome", version: "84"}, {brand: "Chromium", version: "84"} ]
  const mobileness = uaData.mobile; // false
  const platform = uaData.platform; // “macOS”
  (async () => {
    // `getHighEntropyValues()` returns a Promise, so needs to be `await`ed on.
    const highEntropyValues = await uaData.getHighEntropyValues(
      ["platformVersion", "architecture", “bitness”, "model", "uaFullVersion"]);
    const platform = highEntropyValues.platform;               // "macOS"
    const platformVersion = highEntropyValues.platformVersion; //"10_15_4"
    const architecture = highEntropyValues.architecture;       // "x86"
    const bitness = highEntropyValues.bitness;                 // "64"
    const model = highEntropyValues.model;                     // ""
    const uaFullVersion = highEntropyValues.uaFullVersion;     // "84.0.4113.0"
  })();

Use cases

See https://wicg.github.io/ua-client-hints/#use-cases

FAQ

What user needs are solved by reducing the User-Agent header or adding User Agent Client Hints to the web platform?

As mentioned at the beginning of this explainer, we believe that the User-Agent string provides too much passively-available entropy for actors wishing to track users via fingerprinting. By moving the collection of this information to an active model, the user agent—either implicitly on the user’s behalf, or explicitly through browser settings perhaps—can intervene and modify or refuse to provide certain bits of information. This is a privacy win for users.

Do we really need to neuter the JavaScript interface (i.e. Navigator.userAgent) too?

An excellent question! This proposal assumes that developers with access to JavaScript execution do not need the user agent string in order to determine which resources to load and how they ought to behave. They can examine other parts of the exposed API surface (WEBGL_debug_renderer_info, for example). Developers who need this kind of information at request-time could probably migrate to alternative mechanisms like Client Hints.

Our goal should eventually be to ratchet down on some of this granularity as well, and my intuition is that we'll be able to do that more cleanly if we adjust the UA string in one fell swoop, and then move on to the rest rather than doubling back at some point in the future.

What about the compatibility hit we'll take from UA sniffing?

Locking the User-Agent string will lock in the existing behavior of UA-sniffing libraries. That may break things in the future if we diverge significantly from today's behavior in some interesting way, but that doesn't seem like a risk unique to this proposal.

Should the UA string really be a set?

Yes!

History has shown us that there are real incentives for user agents to lie about their branding in order to thread the needle of sites' sniffing scripts, and prevent their users from being blocked by UA-based allow/block lists.

Resetting expectations may help to prevent abuse of the UA string's brand in the short term, but probably won't help in the long run.

Having UA be a set enables browsers to express their brand, as well as their different equivalence sets. We hope that this enabled expressiveness will enable sites to differentially serve based on capabilities while minimizing wrongfully categorizing browsers in the wrong buckets.

Complementary to that, user agents might encourage standardized processing of the UA string by randomly including additional, intentionally incorrect, comma-separated entries with arbitrary ordering (similar conceptually to TLS's GREASE).

Let's examine a few examples:

We'd reflect this value in the navigator.userAgentData.brands attribute, which returns an array of dictionaries containing brand and version.

As a concrete example, what UA string would Chrome on iOS send?

Chrome on iOS (as well as Edge on Android and iOS, Firefox on iOS, and every other non-Safari browser on iOS) are interesting cases in which the underlying browser engine on a given platform doesn't match the engine that the relevant browser built themselves. What should we do in these cases?

There are a few options for the string:

  1. "Chrome"; v="73", ")Friendly-Browsing"; v="99", which has the least entropy, but also sets poor expectations.
  2. "CriOS"; v="73", ")Friendly-Browsing"; v="99" (or "Chrome on iOS", v="73", or similar) which is basically what's sent today, and categorizes the browser as distinct.
  3. "CriOS"; v="73", ")Friendly-Browsing"; v="99", "Safari"; v="12", which is interesting.
  4. ")Friendly-Browsing"; v="99", "Chrome"; v="73", "Safari"; v="12", which is more interesting, as it identifies as Chrome, rather than more cryptic CriOS.

Wait a minute, where is the Client Hints infrastructure specified?

The infrastructure is specified as a separate document, and integration with Fetch and HTML happens there.

What's with the Sec-CH- prefix?

Based on some discussion in w3ctag/design-reviews#320, it seems reasonable to forbid access to these headers from JavaScript, and demarcate them as browser-controlled client hints so they can be documented and included in requests without triggering CORS preflights. A Sec-CH- prefix seems like a viable approach.

How does Sec-CH-UA-Mobile define "mobile"?

This is a tough question. The motivation for the header is that a majority of user-agent header sniffing is used by the server to decide if a "desktop" or "mobile" UX should be served. This is currently implicitly defined in most modern browsers because they have two distinct UIs, a "desktop" version (i.e. Windows, Mac OS, etc.) and a "mobile" version (i.e. Android, iOS). In general, most browsers will also explicitly send "Mobile" in user-agent strings on "mobile" platforms. As such, servers will usually use the platform or presence of this "mobile" identifier. It's also worth pointing out that most modern browsers also have an explicit "request desktop site" UI element in their mobile versions which should be honored. In a more general sense, though, a "mobile" experience could be seen as a UX designed with smaller screens and touch-based interface in mind.

Aren’t we duplicating a lot of information already in the User-Agent header?

It’s true that for some period of time there will be redundancy between the User-Agent header and the hints for platform, browser name and version, and mobileness. Over time, it may be possible to further freeze the User-Agent header to a single static string as the web platform adopts to Client Hints.

Aren’t you adding a lot of new headers? Isn’t that going to bloat requests?

It’s true that we are adding multiple new headers per request. That’s quite a lot. But we don’t expect every site to need all the hints for every request. For the most common requests, compression technologies like HPACK for HTTP/2 and QPACK for HTTP/3 should help optimize these requests. It’s also useful to keep in mind that by default User-Agent Client Hints are only sent to top-level origins by default—explicit delegation is required to send them to third-party origins or embedded frames. We feel the privacy wins and moving away from User-Agent’s legacy are a worthwhile tradeoff here, at the expense of adding some new headers to requests.

Why does a WoW64 hint exist?

WoW64 indicates that a 32-bit User-Agent application is running on a 64-bit Windows machine. It was commonly used to know which NPAPI plugin installer should be offered for download. It's included here for backwards compatibility considerations, to provide a one to one mapping from the UA string of certain browsers to UA-CH. Note that not all browsers today have exposed this, and it may not always return a useful value.

Considered alternatives

Freezing the UA string and reducing its information density without providing an alternative mechanism

We've considered removing information from the UA string, similar to WebKit's attempt.

We ruled out this alternative as it seems to leave many use-cases unaddressed and encourages covert fingerprinting as a means to get at previously-removed information.

Restructuring of the User-Agent string

Restructuring the User-Agent string could have addressed some of the compatibility concerns with today's use of the User-Agent string, and potentially discourage its abuse. At the same time, it would have incurred same or higher migration costs as this proposal. In particular, attempts to restructure the string in-place would result in a lot of breakage of legacy apps. On top of that, it wouldn’t have addressed any of the passive entropy concerns that are motivating this proposal.