[🐛 Bug]: Can not establish connection with Authorization header in Java client since 4.14.0

[Aleks92]

What happened?

We use Java 11 / Selenium 4.12.1 I try to upgrade Selenium, but on version 4.14.0 and higher we can't establish connection with remote webdriver, getting 404. On 4.13.0 all good, don't change anything except version.

I find 2 same issues: https://github.com/SeleniumHQ/selenium/issues/13616 https://github.com/SeleniumHQ/selenium/issues/13802

But we use header Authorization with OAuth token, and not use user/password authorization, so i can't use authenticateAs() method in RemoteWebDriver.builder() Maybe is there some workaround?

How can we reproduce the issue?

private WebDriver initDriver() {
        WebDriver driver;
        URL remoteUrl = config.getRemoteUrl();

        if (Objects.nonNull(remoteUrl)) {
            Callable<RemoteWebDriver> supplier;
            String oAuthToken = config.getOAuthToken();

            HttpClient.Factory factory = config -> {
                HttpClient baseClient = HttpClient.Factory.createDefault().createClient(config.baseUrl());
                return new HttpClient() {
                    @Override
                    public HttpResponse execute(HttpRequest req) throws UncheckedIOException {
                        req.setHeader("Authorization", format("OAuth %s", oAuthToken));
                        HttpResponse response = baseClient.execute(req);
                        return response;
                    }
                    @Override
                    public WebSocket openSocket(HttpRequest request, WebSocket.Listener listener) {
                        return null;
                    }
                };
            };

            HttpCommandExecutor executor = new HttpCommandExecutor(new HashMap<>(), remoteUrl, factory);
            supplier = () -> createRemoteWebdriver(executor, getCapabilities());
            driver = awaitBrowserStart(supplier);

            ((RemoteWebDriver) driver).setFileDetector(new LocalFileDetector());
        } else {
            driver = config.isLocal()
                    ? new ChromeDriver(getChromeOptions().merge(capabilities))
                    : new RemoteWebDriver(remoteUrl, getCapabilities());
        }

        setTimeouts(driver);
        return driver;
    }

    private WebDriver awaitBrowserStart(Callable<RemoteWebDriver> supplier) {
        return await()
                .atMost(config.getBrowserStartTimeout(), TimeUnit.SECONDS)
                .until(supplier, notNullValue());
    }

    private RemoteWebDriver createRemoteWebdriver(CommandExecutor executor, Capabilities capabilities) {
        return new RemoteWebDriver(executor, capabilities);
    }

### Relevant log output

```shell
июн. 27, 2024 7:10:33 PM org.openqa.selenium.remote.RemoteWebDriver log
FINE: Executing: newSession [null, newSession {capabilities=[Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 117.0, goog:chromeOptions: {args: [--disable-web-security, --disable-site-isolation-tr..., --remote-allow-origins=*, --allow-insecure-localhost, --ignore-certificate-errors], excludeSwitches: [enable-automation], extensions: [], mobileEmulation: {deviceMetrics: {height: 731, pixelRatio: 2.6, width: 411}, userAgent: Mozilla/5.0 (Linux; Android...}}, goog:loggingPrefs: org.openqa.selenium.logging..., pageLoadStrategy: eager, platformName: any, selenoid:options: Capabilities {enableLog: tr...}]}]
июн. 27, 2024 7:10:33 PM org.openqa.selenium.remote.http.jdk.JdkHttpClient execute0
FINE: Executing request: (POST) /session
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.HttpClientImpl sendAsync
FINE: [Forwarding newSession on session null to remote] [218ms] HttpClientImpl(1) ClientImpl (async) send http://sw.team.ru:80/wd/v1/quotas/session POST
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Exchange establishExchange
FINE: [pool-3-thread-1] [222ms] Exchange establishing exchange for http://sw.team.ru:80/wd/v1/quotas/session POST,
     proxy=null
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.HttpClientImpl$SelectorManager run
FINE: [HttpClient-1-SelectorManager] [223ms] HttpClientImpl(1) next timeout: 179999
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.HttpClientImpl$SelectorManager run
FINE: [HttpClient-1-SelectorManager] [223ms] HttpClientImpl(1) next expired: 0
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.HttpClientImpl$SelectorManager run
FINE: [HttpClient-1-SelectorManager] [223ms] HttpClientImpl(1) Next deadline is 3000
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http2ClientImpl getConnectionFor
FINE: [pool-3-thread-1] [226ms] Http2ClientImpl not found in connection pool
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ExchangeImpl get
FINE: [pool-3-thread-1] [226ms] ExchangeImpl get: Trying to get HTTP/2 connection
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ExchangeImpl createExchangeImpl
FINE: [pool-3-thread-1] [227ms] ExchangeImpl handling HTTP/2 connection creation result
...
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response$HeadersReader handle
FINE: [pool-3-thread-1] [316ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) Sending 129/129 bytes to header parser
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response$HeadersReader handle
FINE: [pool-3-thread-1] [317ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) Parsing headers completed. bytes=124
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver unsubscribe
FINE: [pool-3-thread-1] [317ms] Http1AsyncReceiver(SocketTube(1)) Unsubscribed jdk.internal.net.http.Http1Response$HeadersReader@17058299
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response lambda$readHeadersAsync$0
FINE: [pool-3-thread-1] [317ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) Reading Headers: creating Response object; state is now READING_BODY
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.common.Utils lambda$wrapForDebug$8
FINE: [pool-3-thread-1] [317ms] Http1Exchange getResponseAsync completed successfully
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response readBody
FINE: [pool-3-thread-1] [318ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) readBody: return2Cache: true
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver onReadError
FINE: [HttpClient-1-SelectorManager] [319ms] Http1AsyncReceiver(SocketTube(1)) recorded java.io.IOException: Connection reset by peer
     delegate: jdk.internal.net.http.Http1Response$HeadersReader@17058299        queue.isEmpty: false
java.io.IOException: Connection reset by peer
    at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
    at java.net.http/jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1153)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:821)
    at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:175)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
    at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:763)
    at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:941)
    at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:245)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:957)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:912)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:912)

июн. 27, 2024 7:10:33 PM jdk.internal.net.http.SocketTube debugState
FINE: [HttpClient-1-SelectorManager] [319ms] SocketTube(1) leaving read() loop with error:  Reading: [ops=0, demand=0, stopped=true], Writing: [ops=0, demand=1]
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription read
FINE: [HttpClient-1-SelectorManager] [319ms] SocketTube(1) Read scheduler stopped
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.HttpClientImpl$SelectorManager run
FINE: [HttpClient-1-SelectorManager] [319ms] HttpClientImpl(1) next timeout: 0
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.HttpClientImpl$SelectorManager run
FINE: [HttpClient-1-SelectorManager] [319ms] HttpClientImpl(1) next expired: 0
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.HttpClientImpl$SelectorManager run
FINE: [HttpClient-1-SelectorManager] [319ms] HttpClientImpl(1) Next deadline is 3000
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response$ClientRefCountTracker acquire
FINE: [pool-3-thread-1] [323ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) Operation started: incrementing ref count for jdk.internal.net.http.HttpClientImpl@67b0c029(1)
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver subscribe
FINE: [pool-3-thread-1] [324ms] Http1AsyncReceiver(SocketTube(1)) Subscribed pending jdk.internal.net.http.Http1Response$BodyReader@29ca67c/parser=jdk.internal.net.http.ResponseContent$ChunkedBodyParser@42ca2c71 queue.isEmpty: false
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver flush
FINE: [pool-3-thread-1] [330ms] Http1AsyncReceiver(SocketTube(1)) Delegate done: 5
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ResponseContent$ChunkedBodyParser onSubscribe
FINE: [pool-3-thread-1] [330ms] PlainHttpConnection(SocketTube(1))/ResponseContent/ChunkedBodyParser onSubscribe: jdk.internal.net.http.Http1Response$Http1BodySubscriber
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver handlePendingDelegate
FINE: [pool-3-thread-1] [330ms] Http1AsyncReceiver(SocketTube(1)) delegate is now jdk.internal.net.http.Http1Response$BodyReader@29ca67c/parser=jdk.internal.net.http.ResponseContent$ChunkedBodyParser@42ca2c71, demand=9223372036854775807, canRequestMore=false, queue.isEmpty=false
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver flush
FINE: [pool-3-thread-1] [330ms] Http1AsyncReceiver(SocketTube(1)) Got 5 bytes for delegate jdk.internal.net.http.Http1Response$BodyReader@29ca67c/parser=jdk.internal.net.http.ResponseContent$ChunkedBodyParser@42ca2c71
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver hasDemand
FINE: [pool-3-thread-1] [330ms] Http1AsyncReceiver(SocketTube(1)) downstream subscription demand is 9223372036854775807
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver flush
FINE: [pool-3-thread-1] [330ms] Http1AsyncReceiver(SocketTube(1)) Forwarding 5 bytes to delegate jdk.internal.net.http.Http1Response$BodyReader@29ca67c/parser=jdk.internal.net.http.ResponseContent$ChunkedBodyParser@42ca2c71
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response$BodyReader handle
FINE: [pool-3-thread-1] [330ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) Sending 5/129 bytes to body parser
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ResponseContent$ChunkedBodyParser tryReadOneHunk
FINE: [pool-3-thread-1] [331ms] PlainHttpConnection(SocketTube(1))/ResponseContent/ChunkedBodyParser Trying to read chunk len (remaining in buffer:5)
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ResponseContent$ChunkedBodyParser tryReadOneHunk
FINE: [pool-3-thread-1] [331ms] PlainHttpConnection(SocketTube(1))/ResponseContent/ChunkedBodyParser Got chunk len 0
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ResponseContent$ChunkedBodyParser tryReadOneHunk
FINE: [pool-3-thread-1] [331ms] PlainHttpConnection(SocketTube(1))/ResponseContent/ChunkedBodyParser Trying to consume bytes: 2 (remaining in buffer: 2)
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ResponseContent$ChunkedBodyParser tryReadOneHunk
FINE: [pool-3-thread-1] [331ms] PlainHttpConnection(SocketTube(1))/ResponseContent/ChunkedBodyParser No more chunks: 0
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ResponseContent$ChunkedBodyParser accept
FINE: [pool-3-thread-1] [331ms] PlainHttpConnection(SocketTube(1))/ResponseContent/ChunkedBodyParser done!
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver clear
FINE: [pool-3-thread-1] [331ms] Http1AsyncReceiver(SocketTube(1)) cleared
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response onFinished
FINE: [pool-3-thread-1] [332ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) SocketTube(1): return to HTTP/1.1 pool
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.PlainHttpConnection close
FINE: [pool-3-thread-1] [332ms] PlainHttpConnection(SocketTube(1)) Closing channel: channel registered with selector, key.interestOps=0, sa.interestOps=0
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.ResponseContent$ChunkedBodyParser accept
FINE: [pool-3-thread-1] [333ms] PlainHttpConnection(SocketTube(1))/ResponseContent/ChunkedBodyParser subscriber completed
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response lambda$readBody$2
FINE: [pool-3-thread-1] [333ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) Finished reading body: READING_BODY
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1Response$ClientRefCountTracker tryRelease
FINE: [pool-3-thread-1] [333ms] Http1Response(id=1, PlainHttpConnection(SocketTube(1))) Operation finished: decrementing ref count for jdk.internal.net.http.HttpClientImpl@67b0c029(1)
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver flush
FINE: [pool-3-thread-1] [334ms] Http1AsyncReceiver(SocketTube(1)) Delegate done: 0
июн. 27, 2024 7:10:33 PM jdk.internal.net.http.Http1AsyncReceiver flush
FINE: [pool-3-thread-1] [334ms] Http1AsyncReceiver(SocketTube(1)) Got 0 bytes for delegate null
июн. 27, 2024 7:10:33 PM org.openqa.selenium.remote.http.jdk.JdkHttpClient execute0
FINE: Ending request (POST) /session in 119ms
__________
REQUEST:
URI - /session
{
  "capabilities": {
    "firstMatch": [
      {
        "acceptInsecureCerts": true,
        "browserName": "chrome",
        "browserVersion": "117.0",
        "goog:chromeOptions": {
          "args": [
            "--disable-web-security",
            "--disable-site-isolation-trials",
            "--remote-allow-origins=*",
            "--allow-insecure-localhost",
            "--ignore-certificate-errors"
          ],
          "excludeSwitches": [
            "enable-automation"
          ],
          "extensions": [
          ],
          "mobileEmulation": {
            "deviceMetrics": {
              "pixelRatio": 2.6,
              "width": 411,
              "height": 731
            },
            "userAgent": "Mozilla\u002f5.0 (Linux; Android 8.0; Pixel 2 Build\u002fOPD3.170816.012) AppleWebKit\u002f537.36 (KHTML, like Gecko) Chrome\u002f106.0.5249.61 Mobile Safari\u002f537.36"
          }
        },
        "goog:loggingPrefs": {
          "browser": "ALL",
          "performance": "INFO"
        },
        "pageLoadStrategy": "eager",
        "platformName": "any",
        "selenoid:options": {
          "shmSize": 1073741824,
          "hosts": [
            "x01.aidata.io:127.0.0.1",
            "static.criteo.net:127.0.0.1"
          ],
          "enableLog": true,
          "enableVideo": false,
          "env": [
            "LANG=ru_RU.UTF-8",
            "LANGUAGE=ru:en",
            "LC_ALL=ru_RU.UTF-8",
            "TZ=Europe\u002fMoscow"
          ]
        }
      }
    ]
  }
}

Headers:
X-Request-Id 2b4b6f7fdaed454586ce994b93789a35
User-Agent selenium/4.14.0 (java mac)
Authorization OAuth y1_AQAD-...
content-type application/json; charset=utf-8

RESPONSE:
Status code - 404

x-request-id 2b4b6f7fdaed454586ce994b93789a35
transfer-encoding chunked

Operating System

macOS Sonoma

Selenium version

4.14.0 / 4.21.0

What are the browser(s) and version(s) where you see this issue?

Chrome 117.0 / 121.0

What are the browser driver(s) and version(s) where you see this issue?

Should ask infra team, it's not displayed

Are you using Selenium Grid?

No response

[github-actions[bot]]

@Aleks92, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[diemol]

A change in the HTTP client changed things, this is how you can do it now.

https://www.selenium.dev/documentation/grid/configuration/toml_options/#basic-auth-enabled