IlegalAccessError throws when using browsermob with WebDriver 2.30.0 to capture n/w

[JaneLee]

Hi , lightbody,

I was so excited after reading a blog about how to capture n/w using WebDriver. One solution is to use browsermob-proxy. But unfortunately I met one issue during my practise project with latest firefox version. And it works not OK etither on IE8 or on Chrome browsers. Causing we need to cover mutliple browsers testing. Pls let me know how to solve this issue! Thx

---

Running environment:

Firefox 19.0.2 WebDriver-Server >= 2.27.0

---

Maven dependency configurations:

<dependencies>
    <dependency>
        <groupId>commons-lang</groupId>
        <artifactId>commons-lang</artifactId>
        <version>2.6</version>
    </dependency>
    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>6.8</version>
    </dependency>
    <dependency>
        <groupId>org.seleniumhq.selenium</groupId>
        <artifactId>selenium-server</artifactId>
        <version>2.30.0</version>
    </dependency>
    <dependency>
        <groupId>biz.neustar</groupId>
        <artifactId>browsermob-proxy</artifactId>
        <version>2.0-beta-7</version>
        <scope>test</scope>
        <exclusions>
            <exclusion>
                <groupId>org.seleniumhq.selenium</groupId>
                <artifactId>selenium-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

</dependencies>

---

My code using browsermob to capture n/w

@Test
// TODO: using Firefox failed
public void testFirefoxWithOBMP() throws Exception {
    // start the proxy
    ProxyServer server = new ProxyServer(5555);
    server.start();

    // get the Selenium proxy object
    Proxy proxy = server.seleniumProxy();

    // configure it as a desired capability
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability(CapabilityType.PROXY, proxy);

    // start the browser up
    System.setProperty("webdriver.firefox.bin", "D:\\Program Files\\Mozilla Firefox\\firefox.exe");
    WebDriver driver = new FirefoxDriver(capabilities);

    // create a new HAR with the label "innerhotsale"
    server.newHar("innerhotsale");

    // open re.taobao.com/search?
    driver.get(INNER_HOTSALE_URL);
    driver.manage().window().maximize();

    // get the HAR data
    Har har = server.getHar();

    System.out.println("============" + har.getLog().getEntries().size());

    System.out.println("END!!");

    server.stop();
}

---

Following are error logs:

INFO 03/13 07:12:09 o.b.p.j.h.HttpServer - Version Jetty/5.1.x INFO 03/13 07:12:09 o.b.p.j.u.Container - Started HttpContext[/,/] INFO 03/13 07:12:09 o.b.p.j.h.SocketLis~ - Started SocketListener on 0.0.0.0:5555 INFO 03/13 07:12:09 o.b.p.j.u.Container - Started org.browsermob.proxy.jetty.jetty.Server@12be1bd WARN 03/13 07:12:23 o.b.p.j.h.HttpConne~ - GET http://re.taobao.com/search? HTTP/1.1 java.lang.IllegalAccessError: tried to access method java.net.InetSocketAddress.getHostString()Ljava/lang/String; from class org.browsermob.proxy.http.SimulatedSocketFactory at org.browsermob.proxy.http.SimulatedSocketFactory.connectSocket(SimulatedSocketFactory.java:118) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:561) at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:372) at org.browsermob.proxy.http.BrowserMobHttpRequest.execute(BrowserMobHttpRequest.java:157) at org.browsermob.proxy.BrowserMobProxyHandler.proxyPlainTextRequest(BrowserMobProxyHandler.java:270) at org.browsermob.proxy.selenium.SeleniumProxyHandler.handle(SeleniumProxyHandler.java:185) at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1509) at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1461) at org.browsermob.proxy.jetty.http.HttpServer.service(HttpServer.java:892) at org.browsermob.proxy.jetty.http.HttpConnection.service(HttpConnection.java:815) at org.browsermob.proxy.jetty.http.HttpConnection.handleNext(HttpConnection.java:981) at org.browsermob.proxy.jetty.http.HttpConnection.handle(HttpConnection.java:832) at org.browsermob.proxy.jetty.http.SocketListener.handleConnection(SocketListener.java:245) at org.browsermob.proxy.jetty.util.ThreadedServer.handle(ThreadedServer.java:357) at org.browsermob.proxy.jetty.util.ThreadPool$PoolThread.run(ThreadPool.java:534) WARN 03/13 07:12:24 o.b.p.j.h.HttpConne~ - GET http://re.taobao.com/favicon.ico HTTP/1.1 java.lang.IllegalAccessError: tried to access method java.net.InetSocketAddress.getHostString()Ljava/lang/String; from class org.browsermob.proxy.http.SimulatedSocketFactory at org.browsermob.proxy.http.SimulatedSocketFactory.connectSocket(SimulatedSocketFactory.java:118) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:561) at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:372) at org.browsermob.proxy.http.BrowserMobHttpRequest.execute(BrowserMobHttpRequest.java:157) at org.browsermob.proxy.BrowserMobProxyHandler.proxyPlainTextRequest(BrowserMobProxyHandler.java:270) at org.browsermob.proxy.selenium.SeleniumProxyHandler.handle(SeleniumProxyHandler.java:185) at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1509) at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1461) at org.browsermob.proxy.jetty.http.HttpServer.service(HttpServer.java:892) at org.browsermob.proxy.jetty.http.HttpConnection.service(HttpConnection.java:815) at org.browsermob.proxy.jetty.http.HttpConnection.handleNext(HttpConnection.java:981) at org.browsermob.proxy.jetty.http.HttpConnection.handle(HttpConnection.java:832) at org.browsermob.proxy.jetty.http.SocketListener.handleConnection(SocketListener.java:245) at org.browsermob.proxy.jetty.util.ThreadedServer.handle(ThreadedServer.java:357) at org.browsermob.proxy.jetty.util.ThreadPool$PoolThread.run(ThreadPool.java:534) ============2 END!! INFO 03/13 07:12:24 o.b.p.j.u.ThreadedS~ - Stopping Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=5555] INFO 03/13 07:12:24 o.b.p.j.h.SocketLis~ - Stopped SocketListener on 0.0.0.0:5555 java.lang.NullPointerException at com.etao.ad.p4p.hotsale.learn.testFirefox.afterClass(testFirefox.java:101)

YOURS, Jane Li

[lightbody]

Jane, I believe the problem is that you're using an old version of Java. We require Java 1.6 and it looks like you're running 1.5. Can you confirm?

[JaneLee]

Hi, lightbody,

My java version is "1.6.0_07".

The browsermob dependency of selenium-api version is 2.20.0. We tested when my project configured with following selenium-server version, browsermob works fine. But not OK with the version larger than 2.26.0. Pls help to confirm whether the error is related to it?

YOURS, Jane

[lightbody]

Interesting. Do you have any unique policies/access rights on the JVM? How are you launching it?

java.lang.IllegalAccessError: tried to access method java.net.InetSocketAddress.getHostString()Ljava/lang/String; from class org.browsermob.proxy.http.SimulatedSocketFactory

On Mar 14, 2013, at 6:21 AM, Jane Li notifications@github.com wrote:

Hi, lightbody,

My java version is "1.6.0_07".

The browsermob dependency of selenium-api version is 2.20.0. We tested when my project configured with following selenium-server version, browsermob works fine. But not OK on the version larger than 2.26.0. pls help to confirm whether the error is related to it?

org.seleniumhq.selenium selenium-server 2.20.0

YOURS, Jane

— Reply to this email directly or view it on GitHub.

[schwartzie]

I'm getting the exact same error using the 2.0-beta-7 standalone REST API. I'm using Java version 1.6.0_26 on a 64-bit Debian Squeeze VM.

[lightbody]

Can you boil it down to a bare bones test case?

On Mar 14, 2013, at 10:23 AM, schwartzie notifications@github.com wrote:

I'm getting the exact same error using the 2.0-beta-7 standalone REST API. I'm using Java version 1.6.0_26 on a 64-bit Debian Squeeze VM.

— Reply to this email directly or view it on GitHub.

[schwartzie]

I didn't get any farther than a bare bones test case myself, so here's a complete rundown of what I did:

Launch proxy server:

$ cd browsermob-proxy-2.0-beta-7/bin/
$ sh browsermob-proxy -port 9090
INFO 03/14 18:14:12 o.b.p.Main           - Starting BrowserMob Proxy version 2.0-beta-7
INFO 03/14 18:14:12 o.e.j.u.log          - jetty-7.3.0.v20110203
INFO 03/14 18:14:12 o.e.j.u.log          - started o.e.j.s.ServletContextHandler{/,null}
INFO 03/14 18:14:12 o.e.j.u.log          - Started SelectChannelConnector@0.0.0.0:9090

Create proxy instance, and add page to HAR:

$ curl -X POST http://localhost:9090/proxy
{"port":9091}
$ curl -X PUT http://localhost:9090/proxy/9091/har

...results in the following output from the server:

INFO 03/14 18:14:31 o.b.p.j.h.HttpServer - Version Jetty/5.1.x
INFO 03/14 18:14:31 o.b.p.j.u.Container  - Started HttpContext[/,/]
INFO 03/14 18:14:31 o.b.p.j.h.SocketLis~ - Started SocketListener on 0.0.0.0:9091
INFO 03/14 18:14:31 o.b.p.j.u.Container  - Started org.browsermob.proxy.jetty.jetty.Server@785e8d7d

Make a request via the proxy:

$ curl -iL -x localhost:9091 http://google.com
HTTP/1.1 500 Internal Server Error
Date: Thu, 14 Mar 2013 18:14:58 GMT
Server: Jetty/5.1.x (Linux/2.6.32-5-xen-amd64 amd64 java/1.6.0_26
Connection: close

...results in the following output from the server:

WARN 03/14 18:14:59 o.b.p.j.h.HttpConne~ - GET http://google.com/ HTTP/1.1
java.lang.IllegalAccessError: tried to access method java.net.InetSocketAddress.getHostString()Ljava/lang/String; from class org.browsermob.proxy.http.SimulatedSocketFactory
    at org.browsermob.proxy.http.SimulatedSocketFactory.connectSocket(SimulatedSocketFactory.java:118)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
    at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:561)
    at org.browsermob.proxy.http.BrowserMobHttpClient.execute(BrowserMobHttpClient.java:372)
    at org.browsermob.proxy.http.BrowserMobHttpRequest.execute(BrowserMobHttpRequest.java:157)
    at org.browsermob.proxy.BrowserMobProxyHandler.proxyPlainTextRequest(BrowserMobProxyHandler.java:270)
    at org.browsermob.proxy.selenium.SeleniumProxyHandler.handle(SeleniumProxyHandler.java:185)
    at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1509)
    at org.browsermob.proxy.jetty.http.HttpContext.handle(HttpContext.java:1461)
    at org.browsermob.proxy.jetty.http.HttpServer.service(HttpServer.java:892)
    at org.browsermob.proxy.jetty.http.HttpConnection.service(HttpConnection.java:815)
    at org.browsermob.proxy.jetty.http.HttpConnection.handleNext(HttpConnection.java:981)
    at org.browsermob.proxy.jetty.http.HttpConnection.handle(HttpConnection.java:832)
    at org.browsermob.proxy.jetty.http.SocketListener.handleConnection(SocketListener.java:245)
    at org.browsermob.proxy.jetty.util.ThreadedServer.handle(ThreadedServer.java:357)
    at org.browsermob.proxy.jetty.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

Retrieve HAR:

$ curl http://localhost:9090/proxy/9091/har
{"log":{"entries":[{"time":0,"request":{"method":"GET","headers":[],"queryString":[],"cookies":[],"bodySize":0,"url":"http://google.com/","httpVersion":"HTTP","headersSize":0},"response":{"content":{"size":0,"mimeType":""},"headers":[],"status":-999,"cookies":[],"bodySize":0,"httpVersion":"HTTP","headersSize":0,"statusText":"NO RESPONSE","redirectURL":""},"startedDateTime":"2013-03-14T18:14:58.926+0000","pageref":"Page 1","cache":{}}],"version":"1.1","browser":{"name":"cURL","version":"7.21.0"},"creator":{"name":"BrowserMob Proxy","version":"2.0"},"pages":[{"id":"Page 1","pageTimings":{},"startedDateTime":"2013-03-14T18:14:44.241+0000","title":""}]}}

Let me know if there's anything else I can provide that will help.

[JaneLee]

HI, lightbody,

To getting more clear with this issue, we tried to test on much more different scenarios.

Those are our testing result:

1. webdriver-server's version 2.30.0 vs browsermob-proxy'version 2.0-beta-6 =====> works fine
2. webdriver-server's version 2.30.0 vs browsermob-proxy'version 2.0-beta-7 =====> throws IlegalAccessError always.

Maybe this result can do some help. Pls help to check.

YOURS, Jane

[schwartzie]

I just tried my test case with 2.0-beta-6, and it worked for me too.

[JaneLee]

will this issue be resolved at 2.0-beta-7 version?

[lightbody]

OK it looks like this all came from this comment:

https://github.com/webmetrics/browsermob-proxy/commit/e4c53e185ead976d09f2abfc77712d4be976f5f7#src/main/java/org/browsermob/proxy/http/SimulatedSocketFactory.java

I'm hoping that @rexhoffman can help us get it resolved. Rex?

[rexhoffman]

Hmm... I was using a mac os x version of java 6 when I wrote this, and didn't experience any security based exceptions/issues. I started a new gig since writing this however and don't have any tests set up around the code.

What os/jvms were used?

I have no need to support java 6 at this point. Just seemed a little sad to force 1.7 for one method. I believe the there were concerns regarding the time constraints of using other methods to determine the host. Though we never discussed caching.

Having done some work configuring policies/building hacks to prevent resource access during testing, this smells like a jvm doing security policy checking in the method.

Java security models are painful at best, and seems to diverge between different jvm providers, so my hack to work around getting access to non-public methods should probably be reverted -- since it has been proven to break with some jvms default policy configurations.

I'm not sure if another work-around would be accepted targeting some other method to calculate the host and potentially cache it.

Lesson learned: don't use reflection to grab non-public methods in the jvm -- they may get tightened up/be tighter in another jvm.

[lightbody]

@JaneLee and @schwartzie - can you get back to @rexhoffman with any info you have about the issue?

[JaneLee]

@rexhoffman @lightbody

My OS is window-xp; And using default jvm configration in Intellij IDEA.

[akosicki]

Just confirming the bug, java.lang.IllegalAccessError is being thrown when trying to use proxy.

Environment:

# browsermob-proxy version: 2.0-beta-7
$ uname -a
Linux aleksander-cuda 3.2.0-39-generic-pae #62-Ubuntu SMP Wed Feb 27 22:25:11 UTC 2013 i686 i686 i386 GNU/Linux
$ java -version
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) Server VM (build 20.14-b01, mixed mode)

Steps to reproduce:

# as in tutotorial
$ sh browsermob-proxy -port 9090 &
...
$ curl -X POST http://localhost:9090/proxy
{"port":9091}
$ curl -x localhost:9091 http://google.com
java.lang.IllegalAccessError: tried to access method java.net.InetSocketAddress.getHostString()Ljava/lang/String; from class org.browsermob.proxy.http.SimulatedSocketFactory
    at org.browsermob.proxy.http.SimulatedSocketFactory.connectSocket(SimulatedSocketFactory.java:118)
...

[hyeung-trulia]

I got the same error on mac with JDK 1.7.

[lightbody]

@rexhoffman What do you suggest we do? Looks like quite a few people are having trouble and I'd like to get this resolved before making the next release.

[rexhoffman]

Definitely surprised by the jdk7 issue on mac as the method was supposed to be publicly accessible in java 7. A security manager could cause it however (limiting access to network method)

As per https://github.com/webmetrics/browsermob-proxy/pull/74

My understand is that we wanted to use that method for performance reasons (to avoid a reverse dns lookup). There was discussion to drop support for java 6 because of the method being package scoped.

We don't have many good options. Tightening java security/desire for a fast method/Desire to support java 6 all seem to conflict.

Personally I'd think taking the performance hit and using

http://docs.oracle.com/javase/6/docs/api/java/net/InetSocketAddress.html#getHostName()is the right way to go.

wdyt?

I'd have no problem doing this tomorrow and putting this issue down. I'll also take another look at the code and write a test for this, but I had used the code without issue after writing. So not expecting to find a simple bug.

On Sun, Apr 7, 2013 at 2:29 PM, Patrick Lightbody notifications@github.comwrote:

@rexhoffman https://github.com/rexhoffman What do you suggest we do? Looks like quite a few people are having trouble and I'd like to get this resolved before making the next release.

— Reply to this email directly or view it on GitHubhttps://github.com/webmetrics/browsermob-proxy/issues/90#issuecomment-16024666 .

[annagrgr]

Hi, I'm getting this error too, 2.0-beta-7, java 6.

[lightbody]

I will get this fixed soon.

On Fri, Apr 19, 2013 at 5:00 AM, annagrgr notifications@github.com wrote:

Hi, I'm getting this error too, 2.0-beta-7, java 6.

— Reply to this email directly or view it on GitHubhttps://github.com/webmetrics/browsermob-proxy/issues/90#issuecomment-16649304 .

[cpuser]

can confirm this error: MacOS X, java version "1.6.0_43", 2.0-beta-7

[lightbody]

I've got it reproduced. Now to fix it. Should be tonight I hope.  — Sent from Mailbox for iPhone

On Mon, Apr 22, 2013 at 3:08 AM, cpuser notifications@github.com wrote:

can confirm this error: MacOS X, java version "1.6.0_43", 2.0-beta-7

Reply to this email directly or view it on GitHub: https://github.com/webmetrics/browsermob-proxy/issues/90#issuecomment-16774920

[davehunt]

I'm also seeing this. Hoping for a beta-8 soon! :)

[lightbody]

I just committed a fix for this. For those of you who can build from master, please try it out and confirm it works for you now. I hope to release beta-8 soon!

[davehunt]

I'm getting failures when attempting to build from master. See http://www.pastebin.mozilla.org/2359519 for details. If you're able to build, could you make a release candidate available? I'm happy to test it.

[lightbody]

@davehunt That was actually a goof on my part. Try updating again and doing the build. Everyone else is encouraged to try this latest build out too, since to fully resolve this bug I upgraded from HttpClient 4.1 to 4.2 and it might introduce some weird artifacts.

[davehunt]

@lightbody beta 8 works for me! :)

[breinjhel]

Hi,

Where can I find beta 8? Im using maven dependency by the way.

Thanks.

[lightbody]

http://search.maven.org/#artifactdetails%7Cnet.lightbody.bmp%7Cbrowsermob-proxy%7C2.0-beta-8%7Cjar

[breinjhel]

Thanks!