Proxy with Selenium

[William-Michael-Russell]

Hello, I am making a common framework which integrates Appium and Selenium together.

I have mitmproxy installed (MAC) and am launching chrome browser. I've included the proxy desired capabilities. desiredCapabilities.addArguments("--proxy-server=localhost:8080");

@Test
public void validateChromeLaunch() {
    List<InterceptedMessage> messages = new ArrayList<InterceptedMessage>();

    MitmproxyJava proxy = new MitmproxyJava("/usr/local/bin/mitmdump", (InterceptedMessage m) -> {
        System.out.println("intercepted request for " + m.requestURL.toString());
        messages.add(m);
        return m;
    });
    try {
        proxy.start();
        WebDriver webDriver = DriverFactory.getDriver();
        webDriver.get("http://google.com");
        System.out.println(webDriver.getCurrentUrl());
        messages.forEach(interceptedMessage ->
                                 System.out.println(interceptedMessage.responseBody
                                 ));
        proxy.stop();
    } catch (IOException | TimeoutException | URISyntaxException | InterruptedException e) {
        e.printStackTrace();
    }
}

I can see the proxy exposing the network requests to google, but I cannot print out the intercepted message, it does not capture the network calls.

Have you successfully tested this with Selenium? This project looks like a great replacement for browsermob.

[selenUser]

I have the same issue with appium, can not get network calls, would be great for some help !

[Jonahss]

Hi @williamrussellajb, what do you mean when you say: I can see the proxy exposing the network requests to google

If you run mitmproxy from the commandline before you run your test, and you run your test without the mitmproxy-java code, does mitmproxy log the requests in the UI in the terminal? That would be how to check that the requests from chromedriver are actually going through the proxy.

[William-Michael-Russell]

Hi Jonahss,

Yes, I can see the network calls in mitmproxy from my terminal, manually started. Likewise, I can see them in my intellij command line when using proxy.start().

I never have any break points hit this line of code here though:

 MitmproxyJava proxy = new MitmproxyJava("/usr/local/bin/mitmdump", (InterceptedMessage m) -> {
        System.out.println("intercepted request for " + m.requestURL.toString());
        messages.add(m);
        return m;
    });

Would you like to make a basic project/example?

[Jonahss]

Hi @williamrussellajb, ok I was just checking about the proxy settings. I have an example project here: https://github.com/cloudgrey-io/appiumpro/blob/master/java/src/test/java/Edition065_Capture_Network_Requests.java

Your use looks like everything is right. Could you include the logs you saw? Could help diagnose the problem.

[spinalk]

@Jonahss , I am having the same issue as @williamrussellajb in which the terminal shows the network requests being captured but the lambda function is never called so I can't store the messages that are being captured

[adamMo]

Hello @spinalk, @williamrussellajb, I spent some time recently with the library and my test case is very simillar to yours, this works for me:

org.openqa.selenium.Proxy proxyProxy = new Proxy();
proxyProxy.setHttpProxy("localhost:8080");
proxyProxy.setSslProxy("localhost:8080");

org.openqa.selenium.remote.DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability("proxy", proxyProxy);

ChromeOptions options = new ChromeOptions();
options.setProxy(proxyProxy);

capabilities.setCapability(ChromeOptions.CAPABILITY, options);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);

WebDriver driver = new ChromeDriver(options);

[Jonahss]

Also @adamMo has made some major improvements recently, so try the newest version with the example above ^

[adamMo]

Hello @spinalk @williamrussellajb I think I finally know what is your issue! You probably doesn't have python websockets module installed. I am currently on fresh OS and I faced:

2019-08-01 23:08:34.249 INFO 30360 --- [ main] i.a.m.MitmproxyJava : Mitmproxy started on port 8890 2019-08-01 23:08:34.255 INFO 30360 --- [ Thread-13] i.a.m.MitmproxyJava : Loading script /tmp/mitmproxy-python-plugin8155624357208443309.py 2019-08-01 23:08:34.255 INFO 30360 --- [ Thread-13] i.a.m.MitmproxyJava : in script /tmp/mitmproxy-python-plugin8155624357208443309.py: No module named 'websockets'

you can fix it by calling pip3 install websockets Later on I will update the library to check and install it automatically.

[seshusangeetha]

@Jonahss I do have the same issue mentioned by the above users with Appium and Eclipse IDE is used. I am able to see the traffic in my console. But none of the messages get intercepted or added. the lambda function is never called so I can't store the messages that are being captured. I have my web sockets installed too if we assume that as @adamMo mentioned the web sockets installation as the issue so I guess that could not be the one causing this issue.

Any Quick help or response would be highly helpful. Thanks

[Jonahss]

So if packets are being sent through mitmproxy but they are not being forwarded to your Java lambda functions, there must be an error in the python plugin which takes the mitm-intercepted packets and forwards them via websocket to the Java library.

We can debug this by running mitmproxy with the Python plugin and see if any errors are logged. Try running mitmdump --anticache -s scripts/proxy.py and then trigger some packets to be sent from the device. Hopefully something will be logged which highlights the issue.

scripts/proxy.py is located inside the mitmproxy-java jar file for mitmproxy-java.

[seshusangeetha]

@Jonahss Yes I guess there is some issue with web sockets. When I run as mentioned with proxy.py, it says that in "scripts/proxy.py: No module named 'websockets' " and then the traffic flow continues. May be this is causing the issue.

I tried with the script in both 1.6.1 and 2.0.1 jars but the same issue exists

Something weird is like when I try pip3 install websockets - it says Requirement already satisfied: websockets in /usr/local/lib/python3.7/site-packages (8.0.2)

The other thing is when I try initialising the proxy it gives the following output where it says web socket server started successfully while it says no web sockets module in the script in next line

[main] INFO io.appium.mitmproxy.MitmproxyJava - Starting mitmproxy on port 8080 [WebSocketSelector-14] INFO org.java_websocket.server.WebSocketServer - websocket server started successfully [Thread-6] INFO io.appium.mitmproxy.MitmproxyJava - Loading script /private//mitmproxy-python-plugin5637973557438151506.py [Thread-6] INFO io.appium.mitmproxy.MitmproxyJava - in script /private//mitmproxy-python-plugin5637973557438151506.py: No module named 'websockets' [Thread-6] INFO io.appium.mitmproxy.MitmproxyJava - Proxy server listening at http://*:8080 [main] INFO io.appium.mitmproxy.MitmproxyJava - Mitmproxy started on port 8080

[Jonahss]

And how did you install mitm proxy on your machine?

[seshusangeetha]

I installed it through brew

[Jonahss]

@seshusangeetha ah, that's the problem then. The readme states:

The install method cannot be a prebuilt binary or homebrew, since those packages are missing the Python websockets module. Install via pip or from source.

[kamil147258369]

@seshusangeetha @spinalk Do You have some news about this problem?

[AnandChaurasia-zz]

@seshusangeetha Any update on this issue?

[Jonahss]

Closing due to inactivity. I think we did solve the original issue.

[adeepakb1]

I had the same issue. I was not not able to intercept network calls using java client but was able to get netowrk call when I run mitmproxy manually from terminal. I had installed mitmproxy using homebrew that why this problem was occuring > i installed itmproxy suing pip and its working fine now.

[dangruonan]

You may get the wrong mitmproxypath. because in my os, there are two mitmdump files to exec.

[ogion]

    proxy = new MitmproxyJava("/usr/local/bin/mitmdump", (InterceptedMessage m) -> {
        System.out.println("intercepted request for " + m.requestURL.toString());
        messages.add(m);
        return m;
    });

this starts mitmproxy itself. you dont need to start mitmproxy in a different terminal. if you start a terminal run before the code, you can not get a result. ı faced this simple mistake, maybe yours is like this.

[yemretat]

Thank you so much, brother! It works, I just closed the terminal which runs mitmweb and it works !