search

MachinePublishers / jBrowserDriver

A programmable, embeddable web browser driver compatible with the Selenium WebDriver spec -- headless, WebKit-based, pure Java
Other
809 stars 143 forks source link

java.lang.NullPointerException: Inflater has been closed #267

Closed jonyt closed 7 years ago

jonyt commented 7 years ago

When upgrading from 0.15.0 to 0.16.0 I get:

 java.lang.NullPointerException: Inflater has been closed
    at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
    at java.util.zip.Inflater.inflate(Inflater.java:257)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
    at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:73)
    at com.machinepublishers.jbrowserdriver.Util.toBytes(Util.java:98)
    at com.machinepublishers.jbrowserdriver.ResponseHandler.handleResponse(ResponseHandler.java:52)
    at com.machinepublishers.jbrowserdriver.StreamConnection.getInputStream(StreamConnection.java:401)
    at com.sun.webkit.network.URLLoader.receiveResponse(URLLoader.java:507)
    at com.sun.webkit.network.URLLoader.doRun(URLLoader.java:164)
    at com.sun.webkit.network.URLLoader.lambda$run$94(URLLoader.java:129)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.webkit.network.URLLoader.run(URLLoader.java:128)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

The code is:

ProxyConfig proxyConfig = new ProxyConfig(ProxyConfig.Type.HTTP, "some.ip", 1234);
        JBrowserDriver driver = new JBrowserDriver(Settings.builder().proxy(proxyConfig).
                timezone(Timezone.AMERICA_NEWYORK).build());
        driver.get("https://www.facebook.com");
        driver.quit();

Note that as far as I can tell this only happens when I use a proxy on https://www.facebook.com.

If I turn on debugging I get:

 -Page-> https://www.facebook.com/ ** {timestamp: 1491239119234, state: page_started, progress: 0.1, error: none, contentType: null, frame: 140684540110032}
 ----->> CONNECT www.facebook.com:443 HTTP/1.1
 ----->> Host: www.facebook.com
 ----->> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_111)
 <<----- HTTP/1.0 200 Connection established
 <<----- Proxy-agent: tinyproxy/1.8.3
 ----->> GET / HTTP/1.1
 ----->> Host: www.facebook.com
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Accept-Encoding: gzip, deflate
 ----->> DNT: 1
 ----->> Connection: keep-alive
 <<----- HTTP/1.1 200 OK
 <<----- P3P: CP="Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p"
 <<----- Expires: Sat, 01 Jan 2000 00:00:00 GMT
 <<----- X-Content-Type-Options: nosniff
 <<----- Strict-Transport-Security: max-age=15552000; preload
 <<----- X-Frame-Options: DENY
 <<----- Cache-Control: private, no-cache, no-store, must-revalidate
 <<----- Pragma: no-cache
 <<----- public-key-pins-report-only: max-age=500; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="q4PO2G2cbkZhZ82+JgmRUyGMoAeozA+BSXVXQWB8XWQ="; report-uri="http://reports.fb.com/hpkp/"
 <<----- X-XSS-Protection: 0
 <<----- Set-Cookie: fr=06nQX5UdV9Nt1qyc2..BY4oDL.lx.AAA.0.0.BY4oDL.AWXOMKaH; expires=Sun, 02-Jul-2017 17:05:15 GMT; Max-Age=7776000; path=/; domain=.facebook.com; httponly
 <<----- Vary: Accept-Encoding
 <<----- Content-Encoding: gzip
 <<----- Content-Type: text/html
 <<----- X-FB-Debug: LLByP5N+1CR2h7/15lgpvytRGvKYEqmp4x//MRhfma/ziX3MFKWsGrcEihBmah8UZJ/8EWtPDCE60YYpLraoqg==
 <<----- Date: Mon, 03 Apr 2017 17:05:15 GMT
 <<----- Transfer-Encoding: chunked
 <<----- Connection: keep-alive
 -Page-> https://www.facebook.com/ ** {timestamp: 1491239121049, state: contenttype_received, progress: 0.1, error: none, contentType: text/html, frame: 140684540110032}
 -Page-> https://www.facebook.com/ ** {timestamp: 1491239121318, state: title_received, progress: 0.30000000000000004, error: none, contentType: text/html, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/ya/r/D3h_VHl0meP.css ** {timestamp: 1491239121318, state: resource_started, progress: 0.0, error: none, contentType: null, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yW/r/mp3hKn7ckuG.css ** {timestamp: 1491239121319, state: resource_started, progress: 0.0, error: none, contentType: null, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yd/r/piY-93iI9Ut.css ** {timestamp: 1491239121319, state: resource_started, progress: 0.0, error: none, contentType: null, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yu/r/lZ86cv9aR90.css ** {timestamp: 1491239121319, state: resource_started, progress: 0.0, error: none, contentType: null, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yZ/r/KzzghyidwOW.css ** {timestamp: 1491239121319, state: resource_started, progress: 0.0, error: none, contentType: null, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yk/r/JYkJTzInaWn.css ** {timestamp: 1491239121319, state: resource_started, progress: 0.0, error: none, contentType: null, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/y6/r/_991G40RCJm.js ** {timestamp: 1491239121320, state: resource_started, progress: 0.0, error: none, contentType: null, frame: 140684540110032}
 ----->> CONNECT static.xx.fbcdn.net:443 HTTP/1.1
 ----->> CONNECT static.xx.fbcdn.net:443 HTTP/1.1
 ----->> CONNECT static.xx.fbcdn.net:443 HTTP/1.1
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_111)
 ----->> CONNECT static.xx.fbcdn.net:443 HTTP/1.1
 ----->> CONNECT static.xx.fbcdn.net:443 HTTP/1.1
 ----->> Host: static.xx.fbcdn.net
 ----->> Host: static.xx.fbcdn.net
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_111)
 ----->> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_111)
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_111)
 ----->> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_111)
 <<----- HTTP/1.0 200 Connection established
 <<----- HTTP/1.0 200 Connection established
 <<----- HTTP/1.0 200 Connection established
 <<----- HTTP/1.0 200 Connection established
 <<----- Proxy-agent: tinyproxy/1.8.3
 <<----- Proxy-agent: tinyproxy/1.8.3
 <<----- Proxy-agent: tinyproxy/1.8.3
 <<----- Proxy-agent: tinyproxy/1.8.3
 <<----- HTTP/1.0 200 Connection established
 <<----- Proxy-agent: tinyproxy/1.8.3
 ----->> GET /rsrc.php/v3/yW/r/mp3hKn7ckuG.css HTTP/1.1
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Accept-Encoding: gzip, deflate
 ----->> DNT: 1
 ----->> Referer: https://www.facebook.com/
 ----->> Connection: keep-alive
 ----->> GET /rsrc.php/v3/yu/r/lZ86cv9aR90.css HTTP/1.1
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Accept-Encoding: gzip, deflate
 ----->> DNT: 1
 ----->> Referer: https://www.facebook.com/
 ----->> Connection: keep-alive
 ----->> GET /rsrc.php/v3/ya/r/D3h_VHl0meP.css HTTP/1.1
 ----->> GET /rsrc.php/v3/yZ/r/KzzghyidwOW.css HTTP/1.1
 ----->> GET /rsrc.php/v3/yd/r/piY-93iI9Ut.css HTTP/1.1
 ----->> Host: static.xx.fbcdn.net
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> Accept-Encoding: gzip, deflate
 ----->> Accept-Encoding: gzip, deflate
 ----->> DNT: 1
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Referer: https://www.facebook.com/
 ----->> DNT: 1
 ----->> Connection: keep-alive
 ----->> Accept-Encoding: gzip, deflate
 ----->> Referer: https://www.facebook.com/
 ----->> DNT: 1
 ----->> Connection: keep-alive
 ----->> Referer: https://www.facebook.com/
 ----->> Connection: keep-alive
 <<----- HTTP/1.1 200 OK
 <<----- Access-Control-Allow-Credentials: true
 <<----- Cache-Control: public,max-age=31536000,immutable
 <<----- Content-Type: text/css; charset=utf-8
 <<----- X-Content-Type-Options: nosniff
 <<----- X-XSS-Protection: 0
 <<----- timing-allow-origin: *
 <<----- Expires: Wed, 28 Mar 2018 15:39:19 GMT
 <<----- Last-Modified: Mon, 01 Jan 2001 08:00:00 GMT
 <<----- Access-Control-Allow-Origin: *
 <<----- Vary: Accept-Encoding
 <<----- Content-Encoding: gzip
 <<----- Content-MD5: iNzJrb8JG89YUOTnSr580Q==
 <<----- X-FB-Debug: U3CGwEzAdF1UQZf1DHFoDWeKbwPBBoh5qDpTzi8otEiE566Lxj1d5nSbgg+5UpkzG/mcaFpaY5YvkH/avyHrXQ==
 <<----- Date: Mon, 03 Apr 2017 17:05:16 GMT
 <<----- Connection: keep-alive
 <<----- Content-Length: 8505
 <<----- HTTP/1.1 200 OK
 <<----- Access-Control-Allow-Credentials: true
 <<----- Cache-Control: public,max-age=31536000,immutable
 <<----- Content-Type: text/css; charset=utf-8
 <<----- X-Content-Type-Options: nosniff
 <<----- X-XSS-Protection: 0
 <<----- timing-allow-origin: *
 <<----- Expires: Thu, 29 Mar 2018 07:59:35 GMT
 <<----- Last-Modified: Mon, 01 Jan 2001 08:00:00 GMT
 <<----- Access-Control-Allow-Origin: *
 <<----- Vary: Accept-Encoding
 <<----- Content-Encoding: gzip
 <<----- Content-MD5: W38RcYYkuViWVwhlG5nqMg==
 <<----- X-FB-Debug: BbBmY3ZjY0GOV8M/isYZKxqEWEPgG1b6x/YT/TW5Foj+ykfzh4mhjyfWEUTcLQgHBkQDRmNObGBQRtd3pWXfNA==
 <<----- Date: Mon, 03 Apr 2017 17:05:16 GMT
 <<----- Connection: keep-alive
 <<----- Content-Length: 26083
 <<----- HTTP/1.1 200 OK
 <<----- Access-Control-Allow-Credentials: true
 <<----- Access-Control-Allow-Origin: *
 <<----- Last-Modified: Mon, 01 Jan 2001 08:00:00 GMT
 <<----- Content-Type: text/css; charset=utf-8
 <<----- X-Content-Type-Options: nosniff
 <<----- Strict-Transport-Security: max-age=15552000; preload
 <<----- X-XSS-Protection: 0
 <<----- public-key-pins-report-only: max-age=500; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="q4PO2G2cbkZhZ82+JgmRUyGMoAeozA+BSXVXQWB8XWQ="; report-uri="http://reports.fb.com/hpkp/"
 <<----- timing-allow-origin: *
 <<----- Cache-Control: public,max-age=31536000,immutable
 <<----- Expires: Thu, 29 Mar 2018 08:56:19 GMT
 <<----- Vary: Accept-Encoding
 <<----- Content-Encoding: gzip
 <<----- Content-MD5: cl295IotOhEM8QQdtBeLmQ==
 <<----- X-FB-Debug: 93xqLHxHnI5+i14RpNuzg5giU+YW0bgEvVxOlfk5sJQgdQMdhXspCeDMom3woHXjsFB4kcqw62zTk3Z1njRMPw==
 <<----- Date: Mon, 03 Apr 2017 17:05:17 GMT
 <<----- Connection: keep-alive
 <<----- Content-Length: 4231
 <<----- HTTP/1.1 200 OK
 <<----- HTTP/1.1 200 OK
 <<----- Access-Control-Allow-Credentials: true
 <<----- Access-Control-Allow-Credentials: true
 <<----- Access-Control-Allow-Origin: *
 <<----- Cache-Control: public,max-age=31536000,immutable
 <<----- Last-Modified: Mon, 01 Jan 2001 08:00:00 GMT
 <<----- Content-Type: text/css; charset=utf-8
 <<----- X-Content-Type-Options: nosniff
 <<----- Content-Type: text/css; charset=utf-8
 <<----- Strict-Transport-Security: max-age=15552000; preload
 <<----- X-Content-Type-Options: nosniff
 <<----- X-XSS-Protection: 0
 <<----- X-XSS-Protection: 0
 <<----- public-key-pins-report-only: max-age=500; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="q4PO2G2cbkZhZ82+JgmRUyGMoAeozA+BSXVXQWB8XWQ="; report-uri="http://reports.fb.com/hpkp/"
 <<----- timing-allow-origin: *
 <<----- timing-allow-origin: *
 <<----- Cache-Control: public,max-age=31536000,immutable
 <<----- Expires: Thu, 29 Mar 2018 22:36:01 GMT
 <<----- Expires: Thu, 29 Mar 2018 10:02:52 GMT
 <<----- Vary: Accept-Encoding
 <<----- Last-Modified: Mon, 01 Jan 2001 08:00:00 GMT
 <<----- Content-Encoding: gzip
 <<----- Access-Control-Allow-Origin: *
 <<----- Content-MD5: wBYH5ciNvfo8IbOrDwDUaQ==
 <<----- Vary: Accept-Encoding
 <<----- X-FB-Debug: bgKixsHmZevTFt4tJANS4aEhByIO3YKfBrfbotvylVbKNaZM6aMxT4k4SxCRY3o8/lH4KkKD2HBr/eqX+QDUDw==
 <<----- Content-Encoding: gzip
 <<----- Date: Mon, 03 Apr 2017 17:05:17 GMT
 <<----- Content-MD5: q06/jCrnX8QzC44kwg2f+Q==
 <<----- Connection: keep-alive
 <<----- Content-Length: 1292
 <<----- X-FB-Debug: OiLbATdX/GqOFuhsz4CQb29yLiidNAi4Z14TaMQuBywCC9vZRUXBRiLEwWx7Jlbc/o1gq2jVLexDxfSOkY9EPA==
 <<----- Date: Mon, 03 Apr 2017 17:05:17 GMT
 <<----- Connection: keep-alive
 <<----- Content-Length: 4646
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yW/r/mp3hKn7ckuG.css ** {timestamp: 1491239123159, state: resource_finished, progress: 1.0, error: none, contentType: text/html, frame: 140684540110032}
 ----->> GET /rsrc.php/v3/yk/r/JYkJTzInaWn.css HTTP/1.1
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Accept-Encoding: gzip, deflate
 ----->> DNT: 1
 ----->> Referer: https://www.facebook.com/
 ----->> Connection: keep-alive
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yZ/r/KzzghyidwOW.css ** {timestamp: 1491239123241, state: resource_finished, progress: 1.0, error: none, contentType: text/html, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/ya/r/D3h_VHl0meP.css ** {timestamp: 1491239123243, state: resource_finished, progress: 1.0, error: none, contentType: text/html, frame: 140684540110032}
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yu/r/lZ86cv9aR90.css ** {timestamp: 1491239123244, state: resource_finished, progress: 1.0, error: none, contentType: text/html, frame: 140684540110032}
 ----->> GET /rsrc.php/v3/y6/r/_991G40RCJm.js HTTP/1.1
 ----->> Host: static.xx.fbcdn.net
 ----->> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
 ----->> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 ----->> Accept-Language: en-us,en;q=0.5
 ----->> Accept-Encoding: gzip, deflate
 ----->> DNT: 1
 ----->> Referer: https://www.facebook.com/
 ----->> Connection: keep-alive
 ----->> Origin: https://www.facebook.com
 -Rsrc-> https://static.xx.fbcdn.net/rsrc.php/v3/yd/r/piY-93iI9Ut.css ** {timestamp: 1491239123244, state: resource_finished, progress: 1.0, error: none, contentType: text/html, frame: 140684540110032}
 <<----- HTTP/1.1 200 OK
 <<----- Access-Control-Allow-Credentials: true
 <<----- Cache-Control: public,max-age=31536000,immutable
 <<----- Content-Type: text/css; charset=utf-8
 <<----- X-Content-Type-Options: nosniff
 <<----- X-XSS-Protection: 0
 <<----- timing-allow-origin: *
 <<----- Expires: Wed, 28 Mar 2018 15:39:18 GMT
 <<----- Last-Modified: Mon, 01 Jan 2001 08:00:00 GMT
 <<----- Access-Control-Allow-Origin: *
 <<----- Vary: Accept-Encoding
 <<----- Content-Encoding: gzip
 <<----- Content-MD5: RypONonwJdw1sa+1kOYkRA==
 <<----- X-FB-Debug: 2k1pEqcUykS356tph6dWRSUaWvFt8AjyI29yIuezNuFKryItZfRfPPlCrW2AUKQQ6BsQonylxdQB11u1Gm01gA==
 <<----- Date: Mon, 03 Apr 2017 17:05:17 GMT
 <<----- Connection: keep-alive
 <<----- Content-Length: 15133
 <<----- HTTP/1.1 200 OK
 <<----- Access-Control-Allow-Credentials: true
 <<----- Cache-Control: public,max-age=31536000,immutable
 <<----- Content-Type: application/x-javascript; charset=utf-8
 <<----- X-Content-Type-Options: nosniff
 <<----- X-XSS-Protection: 0
 <<----- timing-allow-origin: *
 <<----- Expires: Thu, 29 Mar 2018 05:40:13 GMT
 <<----- Last-Modified: Mon, 01 Jan 2001 08:00:00 GMT
 <<----- Access-Control-Allow-Origin: *
 <<----- Vary: Accept-Encoding
 <<----- Content-Encoding: gzip
 <<----- Content-MD5: 9Ld9lP9Jyz7iXx6kYd1epw==
 <<----- X-FB-Debug: RsvmLy+PvvSN2EqYGeNUs6Ofcz8y8hon4EINsCRguWELGboEF0jV67g6DBh7ddDZXI4aypWqU15AfhWWSszeKA==
 <<----- Date: Mon, 03 Apr 2017 17:05:17 GMT
 <<----- Connection: keep-alive
 <<----- Content-Length: 55603

OT: but why the two user agents (Apache-HttpClient/4.5.2 (Java/1.8.0_111) and Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0)?

hollingsworthd commented 7 years ago

Please see #242. Essentially I'm not sure at this point what can be done to fix it.

I can't be sure without looking into it more but I suspect the user agent issue is caused by iframes dynamically being created. It's difficult to cause those to take on custom user agents and such. It might be possible to address this with JNA/JNI code interacting with C++ itself but not sure yet.

Also note v0.17.6 is the latest version although I don't expect it to fix anything for you in this case.

hollingsworthd commented 7 years ago

The main part of your post went over my head at first. Very intertesting it changes from 0.15 to 0.16. That could be useful in fixing this. Thanks!

jonyt commented 7 years ago

Well, if it's any help I got the same exception periodically when trying to fetch Facebook pages with jBrowserDriver 0.14.4. However, this didn't prevent fetching the pages. My hunch is that it happened for requests from the FB chat server. In the issue I opened the exception does prevent fetching the page.