[🐛 Bug]: Chrome and Proxy authentication seems not to work?

[gogonutsss]

What happened?

I try to access a url that needs a corporate proxy with authentication in order to be accessed. Chrome seems to go through proxy, but I get the authentication prompt to input the credentials, that I would expect to be automatically set programmatically via my Proxy object:

See more on the simple example below. Code obviously fails on line 27 as it tries to find an element but actually what I have is the prompt on the sceen above.

How can we reproduce the issue?

import org.openqa.selenium.By;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

public class accessingAurlViaCorporateProxy {

    public static void main(String[] args) {

        System.setProperty("webdriver.chrome.driver", "/pathTo/chromedriver");

        ChromeOptions options = new ChromeOptions();
        Proxy proxy = new Proxy();
        proxy.setSocksUsername("myProxyUsername");
        proxy.setSocksPassword("myProxyPassword");
        proxy.setSslProxy("corporateProxy:port");
        options.setCapability("proxy", proxy);
        options.setAcceptInsecureCerts(true);

        WebDriver driver = new ChromeDriver(options);

        try {
            driver.get("https://urlIamTryingToAccess");
            driver.findElement(By.name("username")).sendKeys("aUsername");
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            driver.quit();
        }
    }
}

Relevant log output

Starting ChromeDriver 95.0.4638.69 (6a1600ed572fedecd573b6c2b90a22fe6392a410-refs/branch-heads/4638@{#984}) on port 38327
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
Nov 15, 2021 4:48:50 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Nov 15, 2021 4:48:50 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
INFO: Found exact CDP implementation for version 95
org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"*[name='username']"}
  (Session info: chrome=95.0.4638.69)
For documentation on this error, please visit: https://selenium.dev/exceptions/#no_such_element
Build info: version: '4.0.0', revision: '3a21814679'
System info: host: 'lbmw3573-ath', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '5.10.10-051010-generic', java.version: '1.8.0_292'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [5be7cbc793417e049d7849b3fc05836f, findElement {using=name, value=username}]
Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 95.0.4638.69, chrome: {chromedriverVersion: 95.0.4638.69 (6a1600ed572fe..., userDataDir: /tmp/.com.google.Chrome.B2llvb}, goog:chromeOptions: {debuggerAddress: localhost:38317}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(manual, ssl=custproxy..., se:cdp: ws://localhost:38317/devtoo..., se:cdpVersion: 95.0.4638.69, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 5be7cbc793417e049d7849b3fc05836f
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
    at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
    at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:53)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:164)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:139)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:559)
    at org.openqa.selenium.remote.ElementLocation$ElementFinder$2.findElement(ElementLocation.java:162)
    at org.openqa.selenium.remote.ElementLocation.findElement(ElementLocation.java:66)
    at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:383)
    at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:375)
    at com.gt.experiment.accessingAurlViaCorporateProxy.main(accessingAurlViaCorporateProxy.java:27)

Operating System

Ubuntu 18

Selenium version

Java 4.0.0

What are the browser(s) and version(s) where you see this issue?

Chrome 95.0.4638.69

What are the browser driver(s) and version(s) where you see this issue?

ChromeDriver 95.0.4638.69

Are you using Selenium Grid?

nope

[github-actions[bot]]

@gogonutsss, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[IlyaGav]

You can add Authentication Handler

var handler = new NetworkAuthenticationHandler()
{
    UriMatcher = _ => true,
    Credentials = new PasswordCredentials(username, password)
};
driver.Manage().Network.AddAuthenticationHandler(handler);

This is C# code, but I think there are analogs in java.

[gogonutsss]

Thanks for this, I dont see something relevant to this in https://www.selenium.dev/selenium/docs/api/java/org/openqa/selenium/WebDriver.Options.html

[diemol]

With Selenium 4, you can use basic auth for this, please check https://www.selenium.dev/documentation/webdriver/bidi_apis/#register-basic-auth

[gogonutsss]

Thanks a lot for the response.

I tried using BIDI and proxy setting, code being like below:

public static void main(String[] args) {

    System.setProperty("webdriver.chrome.driver", "/pathTo/chromedriver");

    ChromeOptions options = new ChromeOptions();
    Proxy proxy = new Proxy();
    proxy.setSocksUsername("myProxyUsername");
    proxy.setSocksPassword("myProxyPassword");
    proxy.setSslProxy("corporateProxy:port");
    options.setCapability("proxy", proxy);
    options.setAcceptInsecureCerts(true);

    WebDriver driver = new ChromeDriver(options);

    try {
        Predicate<URI> uriPredicate = uri -> uri.getHost().contains("https://urlIamTryingToAccess");
        ((HasAuthentication) driver).register(uriPredicate,
                UsernameAndPassword.of("myProxyUsername", "myProxyPassword"));
        driver.get("https://urlIamTryingToAccess");
        driver.findElement(By.name("username")).sendKeys("aUsername");
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        driver.quit();
    }
}

Response is a 407 Proxy authentication error:

Tried also, without setting at all Proxy object as a capability, just by using the bidi approach:

public static void main(String[] args) {

    System.setProperty("webdriver.chrome.driver", "/pathTo/chromedriver");

    ChromeOptions options = new ChromeOptions();
    options.setAcceptInsecureCerts(true);
    WebDriver driver = new ChromeDriver(options);

    try {
        Predicate<URI> uriPredicate = uri -> uri.getHost().contains("https://urlIamTryingToAccess");
        ((HasAuthentication) driver).register(uriPredicate,
                UsernameAndPassword.of("myProxyUsername", "myProxyPassword"));
        driver.get("https://urlIamTryingToAccess");
        driver.findElement(By.name("username")).sendKeys("aUsername");
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        driver.quit();
    }
}

In this case, I ended up with something very odd, I do get to see that driver.get is executed, and I get back an status code OK 200, but just a blank screen, nothing actually loaded and code is ready to go to the next command.

[diemol]

Why are you trying to send driver.findElement(By.name("username")).sendKeys("aUsername");? You should be already logged in.

If you have any questions or need help, join us in the IRC/Slack channel where the community can help you as well.

[gogonutsss]

The final part where I try to find elements to type agaist assumes the page of the application has really loaded - assumes I have passed the corporate proxy - and so I should be in a position to try to login to the application itself with my credentials (in other words this is the real test part). Thanks for checking and pointing for the slack channel too.

[hackeryutu]

RemoteWebDriver not work! class cast exception

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.