proxy web client: support opt-in client hints (http headers)

[GlenDC]

Client hints are sourced from either client request or more likely the emulation data.

By default only the following hints are expected to be allowed:

• Sec-CH-UA
• Sec-CH-UA-Mobile
• Sec-CH-UA-Platform

All other headers should be only allowed if:

• the user defined a special proxy header to opt-in to these headers
• and we can also store in the connection context a list of headers that are seen to be opt-in (by the server response), so follow-up requests on same connection are expected to contain these as well

[GlenDC]

Sec-Fetch headers can always be sent AFAIK.

[GlenDC]

Noted that some CH headers do not follow the sec-ch patterns. The ones I know of as "standard" are:

• Device-Memory
• Width
• Viewport-Width
• DPR
• Save-Data
• Downlink
• ECT
• RTT

[GlenDC]

Todo: read carefully again through the client header RFC to see if we miss some details. As Anti-bot wise I don't think the instructions in this issue cover the desired behaviour 100%.

[GlenDC]

RFC, still in Draft, can be found here: https://wicg.github.io/client-hints-infrastructure/

[GlenDC]

Ok the RFC is not as nice as standard RFC's, but I start to get it.

• Similar to a cookie cache we can create a layer that is attached to the connection context and which keeps track of the client hints requested by the service
• Just as with cookies these hints are attached to the origin, as the storage identifier
• Client hints are sent only over secure connections (even low entropy ones aren't sent)
• Low entropy hints are sent by default over secure connections, most are high entropy client hints, and these are only sent over secure connections when requested
• Fetch API has no control over these headers
• AFAIK one cannot disable these headers from a standard browser (except if one uses extensions or w/e), so best to give these, as there are no settings or so to do it as a user in a normal way

A proxy is however not a regular web client ,and so it might well be that the user of the proxy knows that the server for the given request in a usual flow already would have given a list of headers to accept. As such the allow list written about earlier would essentially allow a request to modify the CH storage, similar to how the response Accept-Ch is allowed to do so.

All client hints, including the ones with non-standard names are mentioned in the above RFC so we can make a static list.

Client hints can also be specified in the HTML of a page... E.g.:

<meta http-equiv="Accept-CH" content="Width, Downlink, Sec-CH-UA" />

This is however only possible if we read the full body and thus I do not think we should do this by default. However it might make sense to have an opt-in flag to allow this to be taken into account as well. It should be opt-in enabled when creating the layer, and optionally it can also be configured to be only done if on top of that the user requests it via some kind of header... As it is a bit expensive to do, compared to just piping the content.

[GlenDC]

Note that these hints also should be considered for caching layers. E.g. Vary: Accept, Width, ECT header has impact on the content served and thus in order to serve appropriate content one has to consider this. #45 .

[GlenDC]

There's also the concept of critical client hints, but that's to do with compatibility of requests, and so we can ignore this.

[GlenDC]

Client hints are still provesional so according to the unconfirmed policy found in https://github.com/hyperium/http/issues/573 it will mean we cannot contribute the client hint headers and ACCEPT-CH to those http/headers crates. These header constants can therefore better be added to rama directly.

[GlenDC]

Tasks to be done:

• add ch module to http
• this will provide "client-hint" headers (can be constants, for which inspiration can be found in https://docs.rs/http/latest/src/http/header/name.rs.html#156-987)
• provide layer/service which:
  • provide a CHCacheStore (that must be part of State), can be added at transport layer
  • we can use an AtomicUsize internally using stuff like https://doc.rust-lang.org/stable/std/sync/atomic/struct.AtomicUsize.html#method.fetch_or and other fetch methods to set and get stuff without it causing overhead or a mutex (can even do so with no strict ordering)
  • should only set headers in secure mode
  • to be figured out is how to configure the modus? Just a mode, or split different responsibilities (deleting existing headers that are wrong, setting desired headers, delete all CH headers in non-secure mode, ...)
  • where do we get data from? Is that just a client hint map that has to be avaialble in Request Context in order for it to be allowed to work? This would allow it to be set via a CH config header, but also by stuff like our UA logic directly in the context...

[GlenDC]

Other requirements / deliverables: good testing + doctest example.