Support emulation of the User-Agent

[sadym-chromium]

it should be possible to set a custom user agent:

1) per browsing context with sub-contexts 2) per user context (requires decision from https://github.com/w3c/webdriver-bidi/issues/775)

The emulation should affect the output of Web APIs observable to the page as well as the network headers for outgoing requests.

[whimboo]

Commands for this could be browsingContext.setUserAgent and browsingContext.getUserAgent. Passing null as user agent should reset the formerly set custom user agent.

[whimboo]

Actually retrieving the current user agent could be done by script evaluation and returning navigation.userAgent. So we probably only need a way to set a custom user agent.

[thiagowfx]

Isn't this a duplicate of https://github.com/w3c/webdriver-bidi/issues/446? cc @OrKoN

[OrKoN]

No, it is not: https://github.com/w3c/webdriver-bidi/issues/446 is about getting the original non-emulated user agent from a browser binary without a need to have browsing contexts.

[mathiasbynens]

I wanted to bring up User-Agent Client Hints. Although it’s a separate feature, it’s somewhat related to User-Agent string emulation, and as a user you’d probably want to emulate both the User-Agent string + the Client Hints accordingly (since otherwise it’s trivial for the target page to detect emulation).

Our options are:

A. include Client Hints support as part of this API B. provide a separate API for Client Hints C. support User-Agent emulation without Client Hints emulation

An example of A would be CDP’s Emulation.setUserAgentOverride which accepts an optional userAgentMetadata param that configures the Client Hints.

[css-meeting-bot]

The Browser Testing and Tools Working Group just discussed Emulation features - User agent override.

The full IRC log of that discussion

<AutomatedTester> topic: Emulation features - User agent override
<AutomatedTester> github: https://github.com/w3c/webdriver-bidi/issues/448
<AutomatedTester> q?
<orkon> q+
<AutomatedTester> ack next
<AutomatedTester> orkon: allows you to set to the custom UA and is related to client hints
<AutomatedTester> ... in CDP you can do UA and client hints in 1 command?
<AutomatedTester> q+
<gsnedders> q+
<jgraham> q+
<AutomatedTester> ... client hints is allows you set up things that can affect the UA
<orkon> https://wicg.github.io/ua-client-hints/
<shs96c> q+
<sadym> q+
<AutomatedTester> AutomatedTester: I see little to no value in being able to change the UA for people. Users will assume that by making Chrome change their UA to be like firefox that it will act like firefox and there might actually be interoperable issues that then exposed where since we are making sure that things work cross browser they should be encouraged to use the relavant browser. E.g. pretending to be a mobile safari in chrome isnt going to
<AutomatedTester> give the same result
<AutomatedTester> ack next
<AutomatedTester> ack next
<AutomatedTester> Sam Sneddon [:gsnedders]: I was going to point out that Mozilla is apposed to UACH? and that Apple has some concerns
<AutomatedTester> ... I think we should avoid doing things that builds on other specs that dont have cross browser buy in
<AutomatedTester> ack next
<AutomatedTester> jgraham: UA stance for Mozilla is negative so we can do this as a chrome specific extension
<AutomatedTester> ... UA Client hints has some privacy concerns. This can be then covered in a chrome extension method
<AutomatedTester> ... on the overall value in doing this is value for people testing their site where the version numbers for example
<orkon> q+
<AutomatedTester> ... there are use cases internally for browser vendors
<AutomatedTester> ... in firefox people can set this in prefs but wont be in a specific tests
<AutomatedTester> q?
<AutomatedTester> ack next
<whimboo> q+
<AutomatedTester> shs96c: UA string makes sense but I am curious if this will be used in network interception
<jgraham> Request interception doesn't affect navigator.userAgent
<gsnedders> s/in a chrome extension method/in a chrome extension method or defined in the UACH spec, slight preference towards the UACH spec because then others can implement it/
<jgraham> q+
<AutomatedTester> ... I have a concern that if we allow this that we are going to encourage people to use a specific browser and wont have a positive impact on a interoperable web
<AutomatedTester> q?
<AutomatedTester> ack next
<AutomatedTester> sadym (IRC): it is way more expensive to test on some devices and I don't think it will encourage people to use the wrong browser
<AutomatedTester> shs96c: I think that having the UA set can be useful but pretending to be a different browser is bad
<AutomatedTester> ack next
<AutomatedTester> orkon
<AutomatedTester> orkon: Let's drop CH as that is out of scope. UA changes can be good for browsers vendors but not for the average tester
<AutomatedTester> ... what is webkits stance on changing this?
<AutomatedTester> Sam Sneddon [:gsnedders]: I think this is fine. we already can do this its just exposing to webdriver
<AutomatedTester> ack next
<whimboo> > Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
<AutomatedTester> whimboo: I wanted to mention setting UA is hard to interpret
<AutomatedTester> ... e.g. using this example it doesn't show what the browser is and can lead to false assumptions on the browser
<AutomatedTester> ack next
<AutomatedTester> ... if we take this into account that we give them presets that they can use
<AutomatedTester> jgraham: I think there are use cases for this. The concerns raised are reasonable. I think its on clients to not overly expose this
<AutomatedTester> ... if we have cases where can create the issues on a desktop instead of a mobile device then its fine but we shouldnt encourage people to use that as the final testing value
<AutomatedTester> ... and to whimboo 's point we should give them something they can edit rather than make it all up and get it all wrong
<AutomatedTester> q?
<orkon> q+

[thiagowfx]

Let's start with C).

From the IRC log above:

orkon: Let's drop CH as that is out of scope. UA changes can be good for browsers vendors but not for the average tester

It seems that we do not want to pursue client hints. @OrKoN could you confirm this?

[thiagowfx]

I am intending to add this command as part of the "browsingContext" module (e.g. instead of a new"emulation" module).

[christian-bromann]

Within the WebdriverIO framework we are using script.addPreloadScript to emulate this Web API, e.g.

await this.scriptAddPreloadScript({
    functionDeclaration: /*js*/`() => {
        Object.defineProperty(navigator, 'userAgent', {
            value: ${JSON.stringify(options)}
        })
    }`
})

Is there any advantage to have this handled through the protocol?

[thiagowfx]

Is there any advantage to have this handled through the protocol?

I don't know, hopefully someone else can answer it. I can only imagine that if it was that easy then it would have been brought up before during the WG meeting.

[OrKoN]

Within the WebdriverIO framework we are using script.addPreloadScript to emulate this Web API, e.g.

await this.scriptAddPreloadScript({
    functionDeclaration: /*js*/`() => {
        Object.defineProperty(navigator, 'userAgent', {
            value: ${JSON.stringify(options)}
        })
    }`
})

Is there any advantage to have this handled through the protocol?

I believe this would not be used for outgoing network requests. While it's possible to solve it via request interception, it is often easier to change it at once.

@thiagowfx Client Hints are completely out of scope for the WebDriver BiDi spec. I recall there were also some concerns from Selenium about adding this feature. Perhaps, we need to discuss again given that we have a draft. In Puppeteer, the feature would allow supporting https://pptr.dev/api/puppeteer.page.setuseragent/.

[OrKoN]

There are two options to move forward with this:

1) Implement client side using preload scripts + request interception

Pros: Does not require spec changes. Cons: Request interception introduces round-trips for each request reducing performance if the use case involves only changing the user agent. Complexity moves to the client.

2) Implement browsingContext.setUserAgent as part of the protocol.

Pros: No performance degradation; simple client implementation.
Cons: Requires extending the spec with something that already can be done by other (although less efficient) means.

[css-meeting-bot]

The Browser Testing and Tools Working Group just discussed Support emulation of the User-Agent.

The full IRC log of that discussion

<AutomatedTester> topic: Support emulation of the User-Agent
<AutomatedTester> github: https://github.com/w3c/webdriver-bidi/issues/448
<AutomatedTester> orkon: this is a feature request I filed
<AutomatedTester> and puppeteer supports it
<AutomatedTester> ... and I am currently leaning towards closing this issue
<AutomatedTester> ... and there is a way that we can implement it
<AutomatedTester> q+
<whimboo> q+
<AutomatedTester> automatedtester: from Selenium point of view we don't need this feature and are ahppy for you to close it
<AutomatedTester> ack next
<AutomatedTester> ack next
<AutomatedTester> whimboo: if your fine then great. we can always see about adding this in the future if there are clients that are porting from CDP and are struggling to amek it work
<AutomatedTester> q?
<AutomatedTester> s/amek/make
<MaksimSadym> present-

[OrKoN]

Closed in favor of initially supporting this feature client-side using preload scripts and request interception. Can revisit if more clients require that.

[whimboo]

From the Playwright team there was also the request to add support for emulating the user agent per user context. Maybe this should be more a general issue given that it affects other emulation features and events as well?

[OrKoN]

I think it would be great to file a separate issue to discuss (generic) per user context configuration for all commands, although we would probably need to specify it per command anyway. I will update the description to mention these details for the user agent emulation.

[whimboo]

I think it would be great to file a separate issue to discuss (generic) per user context configuration for all commands

I filed https://github.com/w3c/webdriver-bidi/issues/775 for that wanted discussion.