Selenium not setting HTTPS proxy

stefano-facchini commented 4 years ago

🐛 Bug Report

The proxy running on localhost:8080 cannot intercept the traffic generated by Selenium. Instead, if I browse manually, the proxy is able to intercept the traffic with no issuse.

-->

To Reproduce

I tried setting the proxy both with the org.openqa.selenium.Proxy object, setting Proxy.setHttpProxy() and Proxy.setSslProxy()

`WebDriver driver;

Proxy proxy = new Proxy(); proxy.setHttpProxy(PROXY_SOCKET); // 127.0.0.1:8080 proxy.setSslProxy(PROXY_SOCKET); // 127.0.0.1:8080

System.setProperty("webdriver.chrome.driver", filePath); ChromeOptions opt = new ChromeOptions(); opt.setCapability("proxy", proxy); opt.setHeadless(false);

driver = new ChromeDriver(opt); `

but also with the command line argument

`WebDriver driver;

System.setProperty("webdriver.chrome.driver", filePath);

ChromeOptions opt = new ChromeOptions(); opt.addArguments("--proxy-server=\"http=" + PROXY_SOCKET + ";https=" + PROXY_SOCKET + "\""); opt.setHeadless(false); driver = new ChromeDriver(opt); `

Detailed steps to reproduce the behavior:

My code is part of a Burp plugin, and Burp itself is the proxy I am using and listening on localhost:8080. The code is this one https://pastebin.com/GEjfT24D and it is triggered by a clicked button. Note that when I remove the HTTPS-related settings, pages are loaded but obviously not intercepted by Burp.

Expected behavior

The proxy should be able to intercept the network traffic, as it does when I am manually surfing Internet.

Test script or set of commands reproducing this issue

There is no script related to this issue, since it times out even loading a simple page.

Environment

OS: Windows 10 Browser: Chrome and Firefox Browser version: Chrome 83.0.4103.97, Firefox 76.0.1 Browser Driver version: ChromeDriver 83.0.4103.39, GeckoDriver 0.26 Language Bindings version: Java 8 and 11 Selenium version: 3.141.59

seanpoulter commented 4 years ago

Hypothesis The browser doesn't accept the Burp Suite HTTPS certificates.

Expected Behaviour When you navigate to an HTTPS page with the HTTPS proxy enabled, you'll probably see a certificate error in the browser. In Chrome, if you click on "Not Secure" in the address bar you can inspect the certificate and confirm it's the certificate from Burp Suite. The proxy is working, but it doesn't match the domain. If you remove or set a breakpoint at the call to driver.quit() I'd expect you to see something like this:

Ideas: If this is the issue, you'll have at least two options:

Trust the one certificate. Create a profile and manage the certificates.
Ignore all untrusted certificates. With Chrome, you can add an argument using the Options to run with --ignore-certificate-errors. For Firefox, it is acceptInsecureCerts.

stefano-facchini commented 4 years ago

Hi, thanks for the reply. This is not my case sadly, because it doesn't even show the certificate error. This is the screen shown when I comment the driver.quit() line, avoiding the browser to be closed.

Immagine

The certificate, as you can see, is not even shown in the Not Secure window. But since I can browse manually with no problem, I assume the certificate is correctly installed in the system. I already tried to use profiles and the insecure certs capabilities you have written but it didn't help.

seanpoulter commented 4 years ago

Hrm. How interesting! We can scratch that off the list of possible causes. Your code looks OK too if you're trusting the certificate.

I'm curious. 🤓 What does Burp Suite look like when the browser hangs like that? Are the network requests intercepted by the Burp Suite proxy and blocked until you decide what to do with them? Here's an example of what I mean. Have a look at the Proxy tab.

Firefox Selenium Java Firefox Burp Suite

Chrome Selenium Java Chrome Burp Suite

--

FYI Your code looks OK. Here's what I ran to generate those. I didn't bother to figure out how to use the DesiredCapabilities with Chrome. 🙈

        Proxy proxy = new Proxy();
        proxy.setHttpProxy("127.0.0.1:8080");
        proxy.setSslProxy("127.0.0.1:8080");

        /*
        FirefoxProfile profile = new FirefoxProfile();
        profile.setAcceptUntrustedCertificates(true);

        FirefoxOptions options = new FirefoxOptions();
        options.setCapability(CapabilityType.PROXY, proxy);
        options.setProfile(profile);

        FirefoxDriver driver = new FirefoxDriver(options);
        */

        /**/
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--ignore-certificate-errors");

        options.setCapability(CapabilityType.PROXY, proxy);

        ChromeDriver driver = new ChromeDriver(options);
        /**/
        driver.manage().window().setPosition(new Point(0, 0));
        driver.manage().window().setSize(new Dimension(1920 / 2, 1200));

        driver.get("https://nytimes.com");

stefano-facchini commented 4 years ago

I added the insecure certificate lines to the code anyway, to avoid issues.

For Chrome opt.addArguments("--ignore-certificate-errors"); For Firefox opt.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS, true);

During my tests the Proxy intercept is off, so the selenium requests are not stopped by Burp. However, If I turn it on, no requests are shown until Selenium times out. Also, Burp GUI is frozen while Selenium tries to load the page, so I cannot switch the view from my extension to the Proxy one. I assume there are no pending requests since the Proxy tab title doesn't become red.

seanpoulter commented 4 years ago

To jump ahead, confirm your proxy IP: Burp Suite > Proxy > Options

How interesting. When I run the following program with the Intercept off in Burp Suite the browser works just fine:

Burp Suite with Intercept off

You've said:

Also, Burp GUI is frozen while Selenium tries to load the page, so I cannot switch the view from my extension to the Proxy one.

This sounds strange to me. That might be a clue. 🤔

What else can we ask to narrow this down?

Does the proxy work when manually configured (docs)?
What happens when you run my code? There are a few subtle differences like:
- the "proxy" vs. CapabilityType.PROXY (should be OK)
- where we find ChromeDriver on the PATH (should be OK)
Have you been able to reproduce this on another system?

What other failure modes are there that we can rule out?

Condition	Result
Burp Suite has not started	Chrome says "No internet"
Proxy is not running	"No internet"
⭐	Wrong proxy IP "127.0.0.2"	Loading spinner until timeout 🤨
Wrong proxy port	"No internet"
Use "localhost" instead of "127.0.0.1"	Works OK

--

Code

This assumes that Java, Maven, and ChromeDriver all installed and available.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.example.selenium</groupId>
  <artifactId>issue-8394</artifactId>
  <version>0.0.0</version>

  <name>issue-8394</name>

  <dependencies>
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-java</artifactId>
      <version>3.141.59</version>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
      <plugins>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

src/main/java/com/example/selenium/Issue8394.java

package com.example.selenium;

import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.Dimension;
import org.openqa.selenium.Point;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.WebDriver;

public class Issue8394
{
    public static void main(String[] args)
    {
        Proxy proxy = new Proxy();
        proxy.setHttpProxy("127.0.0.1:8080");
        proxy.setSslProxy("127.0.0.1:8080");

        ChromeOptions options = new ChromeOptions();
        options.addArguments("--ignore-certificate-errors");
        options.setCapability(CapabilityType.PROXY, proxy);

        ChromeDriver driver = new ChromeDriver(options);
        driver.manage().window().setPosition(new Point(0, 0));
        driver.manage().window().setSize(new Dimension(1920 / 2, 1200));

        driver.get("https://nytimes.com");
    }
}

Run it:

mvn clean compile exec:java -Dexec.mainClass="com.example.selenium.Issue8394"

stefano-facchini commented 4 years ago

This is the Proxy option

proxy opt

The proxy is correctly set and I can browse manually with no problems (with Burp intercepting). This tells me that also the certificate is installed in the correct way.
The code you posted, after adding the System.setProperty(...) line of the WebDrivers, RUNS FINE and the traffic is shown in the Burp proxy history.
I am able to reproduce my issue cloning the repository also on a fresh-installed Ubuntu machine (using same IDE and JDK).

Although, there is one difference between my code and yours: After starting the ChromeDriver successfully, the cmd shows

ChromeDriver was started successfully. giu 11, 2020 10:42:49 AM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: W3C

The last two lines are not printed when I executed my project. Are those two generated by Maven or Selenium?

muhammadumair66 commented 4 years ago

Hello , I am facing the same issue , I am using selenium with driver from one year almost but now when i set proxies and run it. It keeps on spinning or loading the page. So it looks there is some kind of bug but when i ran without proxy page loaded fine. What is the issue ? Can you resolve ? Thanx

diemol commented 4 years ago

@stefano-facchini is this still an issue? Seems @seanpoulter shared all the information and the code used to run it successfully. Why don't you use that code after understanding it and move from there? In any case, this issue seems not related to Selenium at all.

stefano-facchini commented 4 years ago

@diemol I have found a workaround, running the selenium code in another thread so it doesn't get stuck . I can't understand if this issue is related to Selenium or to something else, but I had this happening on three different operating systems in three different computer, but worked fine in the fourth one.

muhammadumair66 commented 4 years ago

Hello , the problem is still comming on asian ip address not other ip address earlier it was working fine !

alien-g-48 commented 4 years ago

Is it possible to run intercept the proxy only with selenium code without using maven. If yes can any one share me the code.

diemol commented 4 years ago

I will close this since the issue looks to be outside of Selenium and the thread went stale as well.

akanshapasricha1 commented 3 years ago

To jump ahead, confirm your proxy IP: Burp Suite > Proxy > Options

How interesting. When I run the following program with the Intercept off in Burp Suite the browser works just fine:

You've said:

Also, Burp GUI is frozen while Selenium tries to load the page, so I cannot switch the view from my extension to the Proxy one.

This sounds strange to me. That might be a clue.

What else can we ask to narrow this down?

Does the proxy work when manually configured (docs)?

What happens when you run my code? There are a few subtle differences like:

the "proxy" vs. CapabilityType.PROXY (should be OK)

where we find ChromeDriver on the PATH (should be OK)

Have you been able to reproduce this on another system?

What other failure modes are there that we can rule out?

Condition Result

Burp Suite has not started Chrome says "No internet"

Proxy is not running "No internet"

star Wrong proxy IP "127.0.0.2" Loading spinner until timeout raised_eyebrow

Wrong proxy port "No internet"

Use "localhost" instead of "127.0.0.1" Works OK

--

Code

This assumes that Java, Maven, and ChromeDriver all installed and available.

pom.xml
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.example.selenium</groupId>
  <artifactId>issue-8394</artifactId>
  <version>0.0.0</version>

  <name>issue-8394</name>

  <dependencies>
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-java</artifactId>
      <version>3.141.59</version>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>
src/main/java/com/example/selenium/Issue8394.java
package com.example.selenium;

import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.Dimension;
import org.openqa.selenium.Point;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.WebDriver;

public class Issue8394
{
    public static void main(String[] args)
    {
        Proxy proxy = new Proxy();
        proxy.setHttpProxy("127.0.0.1:8080");
        proxy.setSslProxy("127.0.0.1:8080");

        ChromeOptions options = new ChromeOptions();
        options.addArguments("--ignore-certificate-errors");
        options.setCapability(CapabilityType.PROXY, proxy);

        ChromeDriver driver = new ChromeDriver(options);
        driver.manage().window().setPosition(new Point(0, 0));
        driver.manage().window().setSize(new Dimension(1920 / 2, 1200));

        driver.get("https://nytimes.com");
    }
}
Run it:
mvn clean compile exec:java -Dexec.mainClass="com.example.selenium.Issue8394"

HEy Same thing happen to my side but My question is just - In crome browser it shown 'Not secure' , It there any way to remove or handle this, So Not secure will not come on browser. Please help me out in this.

seanpoulter commented 3 years ago

Hopefully you found the docs last year: https://portswigger.net/burp/documentation/desktop/getting-started/proxy-setup/certificate

alien-g-48 commented 3 years ago

Can I get sample code to invoke burp scanner and spiderthe URL in java.

On Sat, 5 Sep 2020 at 5:07 AM, Diego Molina notifications@github.com wrote:

I will close this since the issue looks to be outside of Selenium and the thread went stale as well.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SeleniumHQ/selenium/issues/8394#issuecomment-687470480, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDLTK5GWJHBJ37UJY6GSYTSEF24NANCNFSM4NWHR7NQ .

SeleniumHQ / selenium