Support for specifying extra HTTP headers

[whimboo]

It would be helpful for clients to have the ability to specify extra HTTP headers globally or for a specific set of browsing contexts. While custom HTTP headers can currently be set through network interception, it would simplify the process if we allowed headers to be set via a single command without relying on interception. These headers could then be automatically applied to all network activity within the specified browsing contexts.

This enhancement would enable more flexible testing scenarios, where each context can simulate different client headers, such as language settings (see #774), authentication tokens, or custom headers for API requests.

Use Cases

1. Multi-Context Testing: Simulating different user agents, authentication tokens, or localization settings (e.g., Accept-Language) across various browsing contexts. This would help developers test how their application behaves when accessed by different types of clients in parallel.

2. A/B Testing: Using different headers (e.g., feature flags) to simulate A/B testing environments within the same browser instance, but isolated to separate browsing contexts.

3. Authentication and Security Testing: Simulating different users with different authentication headers in separate tabs or frames. This allows the testing of security mechanisms and session isolation between tabs. This is some feature Cypress would benefit from given that it has to send a fixed set of authentication headers.

Proposal

Introduce an API that allows setting extra HTTP headers globally or for specific browsing contexts. The API would accept a list of context IDs along with the headers to be applied for each context.

The API could look like:

• Command to set extra HTTP headers for multiple contexts:

  {
    "method": "network.addExtraHTTPHeaders",
    "params": {
      "contexts": ["context1", "context2"],  // The IDs of the browsing contexts
      "headers": {
        "X-Custom-Header": "exampleValue",
        "Accept-Language": "fr-FR"
      }
    }
  }

• Command to clear extra HTTP headers:

  {
    "method": "network.removeExtraHTTPHeaders",
    "params": {
      "contexts": ["context1", "context2"]   // The IDs of the browsing contexts
    }
  }

Benefits

• Granular Control Over HTTP Requests: Developers can test applications with different HTTP headers per context in parallel, making it easier to simulate real-world usage with different users, agents, or environments.
• Multi-Session Testing: Allows the simulation of multiple sessions, users, or clients by attaching different headers to each context, useful for testing session management, security, and caching behavior.
• Localization and Customization: This feature would be particularly helpful for testing internationalization (Accept-Language header) or client-specific behavior based on custom headers.

Questions

• Should there be a way to query the currently active headers for a given context?
• Should we allow removing of individual headers if the headers field and entries are specified?

[mxschmitt]

Note: We've also seen requests from customers for setting multiple headers with the same header name.

[Lightning00Blade]

@whimboo Considering we had discussion around how session.subscribe/session.unsubscribe and how it can become very confusing (here we may hit this as well), I would think about using the same logic we do for script.AddPreloadScript and network.addInterception where we return an ID to be used for the removal operation.

[css-meeting-bot]

The Browser Testing and Tools Working Group just discussed Emulation - Support for specifying extra HTTP headers.

The full IRC log of that discussion

<AutomatedTester> topic: Emulation - Support for specifying extra HTTP headers
<AutomatedTester> github: https://github.com/w3c/webdriver-bidi/issues/777
<AutomatedTester> jgraham: this is an extra command to alter extra HTTP Headers
<AutomatedTester> ... I was hoping to get away with not doing this but it's isomorphic to adding this to network interception
<sadym> q+
<AutomatedTester> ... there are ways that we can do that is described in the issues
<gsnedders> q+
<simonstewart> q+
<AutomatedTester> ... there is an question around handling existing headers
<AutomatedTester> ... so do we overwrite or append?
<AutomatedTester> ack sadym
<AutomatedTester> sadym: my question is around the spec of this. Can we actually spec it?
<AutomatedTester> jgraham: yes, we can reuse the netwrok interception section
<AutomatedTester> ... there is an interesting use case for handling this if you have these set in the network interception and you have set the value
<AutomatedTester> ... there should be infra for this
<AutomatedTester> sadym: there is a question to have a remove the headers later?
<AutomatedTester> jgraham: yes we should have that
<AutomatedTester> q?
<AutomatedTester> ack gsnedders
<AutomatedTester> gsnedders: on the question of multiple headers? I think in reality there have been 3 different sematics for headers...
<dmurph> present-
<AutomatedTester> ... e.g. with duplication that it takes `HEADER: Value1, Value2`
<AutomatedTester> ... and set cookie headers do some magic
<jgraham> q+
<AutomatedTester> ... so I am not sure we can come up with what we believe the sematics will be here
<AutomatedTester> ack simonstewart
<AutomatedTester> simonstewart: I think the logical order is to have set headers and then allow network interception to overwrite
<gsnedders> i/headers do some magic/... first/last (I forget) header for a given name wins
<gsnedders> s/we can come up with what /we can come up with a singular answer of what /
<jimevans> q+
<AutomatedTester> ... it would be good to have a replace headers so that we don't have to worry about the heuristics of how these things are added over time to different specs
<AutomatedTester> ack jgraham
<AutomatedTester> jgraham: I don't think it will work to append
<AutomatedTester> ... there are definitely use cases for replace more than append
<AutomatedTester> ... we can have the default to override except for cookie where it appends
<AutomatedTester> ... or we can have a mode that says append or override
<gsnedders> q+
<AutomatedTester> ack jimevans
<AutomatedTester> jimevans: the syntax for this is going to complex... combining override and append
<jgraham> I am imagining `[{name: "X-Header", value: "foo", mode: "append"}]` with `mode` being optional
<AutomatedTester> ... manipulating all network intercepts with one command is a little hard for to visualise
<AutomatedTester> q?
<AutomatedTester> ack gsnedders
<gsnedders> https://fetch.spec.whatwg.org/#headers-class
<AutomatedTester> gsnedders: I think it would be good to have a look at the Fetch API does for the developers
<AutomatedTester> jgraham: let's do what they do...done
<jgraham> So "append", "delete" and "set" as possible modes
<AutomatedTester> jgraham: I would prefer 1 command
<AutomatedTester> simonstewart: and I think we need to make sure that we have removed any form of ambiguity
<jimevans> scribe+
<dotproto> +q
<dotproto> -q