Closed pschaeffer closed 5 years ago
joakime
commented
5 years ago What version of Jetty? How are you starting Jetty? (be detailed) What OS? (be detailed, versions, updates, kernels, etc..) What JVM? (again, be detailed, versions, 32/64-bit, build identifiers, etc) Since you are using HTTP/2, ALPN will be important, so we'll want to triage your ALPN setup as well (but that depends on knowledge about your JVM)?
joakime
commented
5 years ago What does the request and response headers look like when you have HTTP/2 disabled on Jetty (and are using just straight HTTP/1.1)?
I ask, because Chrome will fail with -337 (ERR_SPDY_PROTOCOL_ERROR) if you are not strictly following the up to date RFCs related to HTTP headers (CORS, Cookie, Cache, etc).
If we can see your raw HTTP/1.1 request / response headers then we can point to problems you might have. You can do this yourself as well, look at the net-internals events when you attempt to connect to Jetty with HTTP/2. If you see something like ...
t=65422 [st=53] HTTP_TRANSACTION_READ_HEADERS [dt=4]
--> net_error = -337 (ERR_SPDY_PROTOCOL_ERROR) ... then you have a bad HTTP header that is causing your issue.
pschaeffer
commented
5 years ago Mr. Joakime,
The Jetty version is jetty-distribution-9.4.18.v20190429. I am running Jetty as an embedded server. I can provide the Java program that starts Jetty, if need be. The OS is Ubuntu Linux 18.04 LTS. Note that the same problem can be exactly reproduced running the Java code under Eclipse under Windows 10. java -version returns java version "11.0.3" 2019-04-16 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.3+12-LTS) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.3+12-LTS, mixed mode) I think ALPN is OK. The following messages appear at Jetty startup.
pschaeffer
commented
5 years ago Dear Sir,
See 6 through 11 below.
I am having a problem using Chrome with Jetty. At the outset, I must say that I am not sure (at all) if this is a Chrome problem, a Jetty problem, or some error on my part. The symptom is easy to describe. Some HTTP(2) GET requests fail with a -337 error. This is described as a SPDY_PROTOCOL_ERROR in some places. The correct equate is probably HTTP2_PROTOCOL_ERROR. A few notes.
If I turn off HTTP2 in Jetty, the problem goes away. The problem only shows up with HTTP2 activated.
The problem can not be reproduced with Firefox. This could be evidence that Chrome has a bug. However, it could be that Firefox just hides the problem better. Or some other difference between Chrome and Firefox may explain the behavior difference.
If I use a PHP based server (Apache 2 with PHP), the problem can not be reproduced. This was a big mystery for a while. Then I found out that my Apache 2 server does not have HTTP2 configured.
If I bypass Jetty and the Apache 2 / PHP server, the problem can not be reproduced. This is strange. However, a check of the Wireshark log explained (explains) this behavior. If both Jetty and Apache 2 / PHP are out of the picture, the files (that cause the problem) are obtained with an HTTP GET, not an HTTP2 GET.
A Chrome network log file (zipped) is attached. You can view this file with (https://netlog-viewer.appspot.com/#import). The failures are events 3294, 3295, and 3300. The Chrome network log describes these problems as ERR_SPDY_PROTOCOL_ERROR with a code of -337. These are probably really HTTP2_PROTOCOL_ERRORs. A possibly useful note is that the Chrome network log shows the expected outbound HTTP headers, but no reply headers.
A Wireshark log (zipped) is attached. You can view this file with Wireshark. The failure(s) start around packet number 890. Note that the client system (Chrome) is 192.168.1.99. The Jetty system is at 3.14.54.178. A good display filter is (ip.src==3.14.54.178 or ip.dst==3.14.54.178). Take a look at packet 1018 (HTTP2 RST_STREAM). This packet may be related to the error.
The Jetty version is jetty-distribution-9.4.18.v20190429.
Jetty is running as an embedded server. I can provide the Java program that runs Jetty, if need be.
The OS is Ubuntu Linux 18.04 LTS. Note that the same problem can be exactly reproduced running the Java code under Eclipse under Windows 10.
java –version returns
java version "11.0.3" 2019-04-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.3+12-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.3+12-LTS, mixed mode)
I think ALPN is OK. The following messages appear at Jetty startup.
Logging initialized @1656ms to org.eclipse.jetty.util.log.Slf4jLog
jetty-9.4.18.v20190429; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 11.0.3+12-LTS
Started o.e.j.s.ServletContextHandler@53fd0d10{/,null,AVAILABLE}
Started ServerConnector@1722011b{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80}
x509=X509@6aa3a905(headlamp,h=[hostname1.com, hostname2.com, hostname3.com, hostname4.com, hostname5.com, hostname6.com, hostname7.com,w=[]) for Server@a22cb6a[provider=null,keyStore=null,trustStore=null]
Started ServerConnector@258d79be{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443}
Started @2047ms
Of course, I can create any other logs or traces that might help resolve this problem. I can also run Jetty under Eclipse and set breakpoints as need be.
Thank you
Peter Schaeffer
From: Joakim Erdfelt [mailto:notifications@github.com] Sent: Wednesday, July 24, 2019 4:51 PM To: eclipse/jetty.project Cc: Peter Schaeffer; Author Subject: Re: [eclipse/jetty.project] Chrome / Jetty Problems (#3904)
What version of Jetty? How are you starting Jetty? (be detailed) What OS? (be detailed, versions, updates, kernels, etc..) What JVM? (again, be detailed, versions, 32/64-bit, build identifiers, etc) Since you are using HTTP/2, ALPN will be important, so we'll want to triage your ALPN setup as well (but that depends on knowledge about your JVM)?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/jetty.project/issues/3904?email_source=notifications&email_token=AA4LSSXLBA4CPYMDUMANOSTQBDFDFA5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XW2PA#issuecomment-514813244 , or mute the thread https://github.com/notifications/unsubscribe-auth/AA4LSSTJCHXYE6ITR5O6RM3QBDFDFANCNFSM4IGUYAXA . https://github.com/notifications/beacon/AA4LSSTMEBONC4W43RSQTPDQBDFDFA5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XW2PA.gif
joakime
commented
5 years ago ALPN on Java 11.0.3 is good choice, it will keep you from having to do the Xbootclasspath stuff with alpn-boot that Java 8 requires.
But since you are using Java 11.0.3, you might want to consider disabling TLSv1.3 (which isn't quite stable yet on Java, and chrome supports it) See: https://webtide.com/openjdk-11-and-tls-1-3-issues/
Have you collected your request / response headers on Jetty without HTTP/2 enabled yet? See: https://github.com/eclipse/jetty.project/issues/3904#issuecomment-514814774
pschaeffer
commented
5 years ago Mr. Joakim Erdfelt,
See below. Of course, I can provide the entire Wireshark file if that would be helpful.
Thank you
Peter Schaeffer
Outbound HTTP headers
Reply HTTP headers
From: Joakim Erdfelt [mailto:notifications@github.com] Sent: Wednesday, July 24, 2019 5:50 PM To: eclipse/jetty.project Cc: Peter Schaeffer; Author Subject: Re: [eclipse/jetty.project] Chrome / Jetty Problems (#3904)
ALPN on Java 11.0.3 is good choice, it will keep you from having to do the Xbootclasspath stuff with alpn-boot that Java 8 requires.
But since you are using Java 11.0.3, you might want to consider disabling TLSv1.3 (which isn't quite stable yet on Java, and chrome supports it) See: https://webtide.com/openjdk-11-and-tls-1-3-issues/
Have you collected your request / response headers on Jetty without HTTP/2 enabled yet? See: #3904 (comment) https://github.com/eclipse/jetty.project/issues/3904#issuecomment-514814774
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/jetty.project/issues/3904?email_source=notifications&email_token=AA4LSSQSY4AYXRXRG4DWSFTQBDL73A5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2X2MRA#issuecomment-514827844 , or mute the thread https://github.com/notifications/unsubscribe-auth/AA4LSSUODB6DEJ6BXUHZ7T3QBDL73ANCNFSM4IGUYAXA . https://github.com/notifications/beacon/AA4LSSUIZWOCBMJDBLHNLU3QBDL73A5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2X2MRA.gif
pschaeffer
commented
5 years ago Mr. Joakim Erdfelt,
My Email below contains a error. Yes, the outbound HTTP headers are correct. However, the reply headers are for an entirely different request.
I can not find the reply headers at this point (which may be the problem).
Thank you
Peter Schaeffer
From: Peter Schaeffer [mailto:peter_schaeffer@yahoo.com] Sent: Wednesday, July 24, 2019 6:08 PM To: 'eclipse/jetty.project'; 'eclipse/jetty.project' Cc: 'Author' Subject: RE: [eclipse/jetty.project] Chrome / Jetty Problems (#3904)
Mr. Joakim Erdfelt,
See below. Of course, I can provide the entire Wireshark file if that would be helpful.
Thank you
Peter Schaeffer
Outbound HTTP headers
Reply HTTP headers
From: Joakim Erdfelt [mailto:notifications@github.com] Sent: Wednesday, July 24, 2019 5:50 PM To: eclipse/jetty.project Cc: Peter Schaeffer; Author Subject: Re: [eclipse/jetty.project] Chrome / Jetty Problems (#3904)
ALPN on Java 11.0.3 is good choice, it will keep you from having to do the Xbootclasspath stuff with alpn-boot that Java 8 requires.
But since you are using Java 11.0.3, you might want to consider disabling TLSv1.3 (which isn't quite stable yet on Java, and chrome supports it) See: https://webtide.com/openjdk-11-and-tls-1-3-issues/
Have you collected your request / response headers on Jetty without HTTP/2 enabled yet? See: #3904 (comment) https://github.com/eclipse/jetty.project/issues/3904#issuecomment-514814774
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/jetty.project/issues/3904?email_source=notifications&email_token=AA4LSSQSY4AYXRXRG4DWSFTQBDL73A5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2X2MRA#issuecomment-514827844 , or mute the thread https://github.com/notifications/unsubscribe-auth/AA4LSSUODB6DEJ6BXUHZ7T3QBDL73ANCNFSM4IGUYAXA . https://github.com/notifications/beacon/AA4LSSUIZWOCBMJDBLHNLU3QBDL73A5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2X2MRA.gif
pschaeffer
commented
5 years ago Mr. Joakim Erdfelt,
I have done some more research into this. There are no reply headers. However, the reason is strange. I am getting an exception in the following code
try {
response.getOutputStream().write(httpResponse.getContent());
}
catch (Exception e) {
LOG.info(requestOriginalURI);
LOG.info("exception");
LOG.info(e.getMessage(), e);
}
The write operation is failing with an exception. A log file with the exception is attached. Note that this is with HTTP2 enabled.
Thank you
Peter Schaeffer
pschaeffer
commented
5 years ago Mr. Joakim Erdfelt,
This time I ran with HTTP2 disabled. See the attached log file for the exception.
Thank you
Peter Schaeffer
From: Peter Schaeffer [mailto:peter_schaeffer@yahoo.com] Sent: Wednesday, July 24, 2019 7:51 PM To: 'eclipse/jetty.project'; 'eclipse/jetty.project' Cc: 'Author' Subject: RE: [eclipse/jetty.project] Chrome / Jetty Problems (#3904)
Mr. Joakim Erdfelt,
I have done some more research into this. There are no reply headers. However, the reason is strange. I am getting an exception in the following code
try {
response.getOutputStream().write(httpResponse.getContent());
}
catch (Exception e) {
LOG.info(requestOriginalURI);
LOG.info("exception");
LOG.info(e.getMessage(), e);
}
The write operation is failing with an exception. A log file with the exception is attached. Note that this is with HTTP2 enabled.
Thank you
Peter Schaeffer
joakime
commented
5 years ago This time I ran with HTTP2 disabled. See the attached log file for the exception.
Please use the github interface when attaching files. Using email replies will not work.
dwoznicki
commented
5 years ago Hi, I'm also experiencing this problem. I'm pretty well out of my wheelhouse with this, but I'll try my best to provide details.
Jetty version: 9.4.19.v20190610
Java version:
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+7-Ubuntu-1ubuntu218.04.1)
OpenJDK 64-Bit Server VM (build 11.0.3+7-Ubuntu-1ubuntu218.04.1, mixed mode, sharing)OS:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
$ uname -a
Linux Cobbler 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/LinuxHTTP/2 setup:
SslContextFactory sslContextFactory = new SslContextFactory();
sslContextFactory.setKeyStorePath("abc");
sslContextFactory.setKeyStorePassword("abc");
sslContextFactory.setExcludeProtocols("TLSv1.3");
sslContextFactory.setCipherComparator(HTTP2Cipher.COMPARATOR);
sslContextFactory.setUseCipherSuitesOrder(true);
HTTP2ServerConnectionFactory h2 = new HTTP2ServerConnectionFactory(httpsConf);
ALPNServerConnectionFactory alpn = new ALPNServerConnectionFactory();
ServerConnector httpsConnector = new ServerConnector(server, new SslConnectionFactory(sslContextFactory, alpn.getProtocol()), alpn, h2, new HttpConnectionFactory(httpsConf));
// httpsConnector is eventually added to web server via setConnectors()Jetty log: jetty-debug.log Chrome net-export log: HTTP2_SESSION-chrome-net-export.log The important part of the Chrome log may be these lines (though I'm not sure):
t=171914 [st=105514] HTTP2_SESSION_CLOSE
--> description = "Framer error: 2 (INVALID_CONTROL_FRAME)."
--> net_error = -337 (ERR_SPDY_PROTOCOL_ERROR)
sbordet
commented
5 years ago I don't see anything wrong on the server logs, it appears to me that this is a Chrome issue. Does it work with other browsers? What exact Chrome version?
joakime
commented
5 years ago @sbordet the attached HTTP2_SESSION-chrome-net-export.log in @dwoznicki comment shows the following ...
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36It works fine in Firefox. I was seeing these net::ERR_SPDY_PROTOCOL_ERROR errors in Chrome sporadically until this morning. Now I'm getting them perhaps 1 in every 10 page refreshes, and I have to refresh another ~10 times before the page loads properly again.
I haven't updated Chrome recently, but I suppose some browser patch might have been applied?
sbordet
commented
5 years ago @dwoznicki I have Chrome/75.0.3770.142 (yours is Chrome/75.0.3770.100) on Ubuntu 19.04 and works fine for me against known HTTP/2 sites. Seems really a Chrome issue to me. Can you update to ...142 and retry?
dwoznicki
commented
5 years ago @sbordet Ok, I just updated to Chrome/75.0.3770.142 and here's what I found:
Updating Chrome seems to have solved the annoying frequency of the errors, so I'm pretty much content at this point.
pschaeffer
commented
5 years ago sbordet,
The problem only shows up with Chrome. With Firefox everything looks OK (at least superficially).
pschaeffer
commented
5 years ago Mr. Joakim Erdfelt,
I have done some more research into this. There are no reply headers. However, the reason is strange. I am getting an exception in the following code
try {
response.getOutputStream().write(httpResponse.getContent());
}
catch (Exception e) {
LOG.info(requestOriginalURI);
LOG.info("exception");
LOG.info(e.getMessage(), e);
}
The write operation is failing with an exception. A log file with the exception is attached. Note that this is with HTTP2 enabled.
Thank you
Peter Schaeffer HDLmLog.log
pschaeffer
commented
5 years ago Mr. Joakim Erdfelt,
This time I ran with HTTP2 disabled. See the attached log file for the exception.
Thank you
Peter Schaeffer HDLmLog.log
sbordet
commented
5 years ago @pschaeffer the HTTP/2 logs show that the server-side write failed because the client reset the stream.
The problem is to understand why the client reset the stream, and so far it appears to be a Chrome issue.
pschaeffer
commented
5 years ago sbordet,
Yes, for some reason the client (Chrome) is resetting the stream Why is not clear. So far Google has not provided any information that might answer this question. Note that if I turn HTTP2 off (and only use HTTP), problems (different problems) still arise. Interestingly enough, these problem show up with both Chrome and Firefox. See the attach log. The server log has both Firefox (toward the beginning) and Chrome problems (toward the end).
Thank you
Peter Schaeffer HDLmLog.log
joakime
commented
5 years ago Perhaps the reload/refresh (that you mentioned) is resetting the prior / active stream? and not the current one you refreshed to?
pschaeffer
commented
5 years ago Hi, I was able to fix my HTTP problems by fixing a bug in my code. However, I am still having HTTP2 problems. Chrome is issuing a reset (why is not clear) that is causing problems with HTTP2. It is not clear if the resets coming from Chrome are a Chrome bug, a Jetty bug, or a bug in my code. Thank you Peter Schaeffer
pschaeffer
commented
5 years ago sbordet, I am running Chrome Version 75.0.3770.142 (Official Build) (64-bit). Thank you Peter Schaeffer
pschaeffer
commented
5 years ago Hi, I finally found the actual problem. Not a bug in Jetty or Chrome as it turns out. The actual bug was in my code. The HTTP2 specification does not allow the server to send a "Transfer-Encoding" header to the client. The Chrome code found a "Transfer-Encoding" header and called ResetStream() in response. The "Transfer-Encoding" header was added by my code (indirectly and inadvertently). I removed the "Transfer-Encoding" header and the problem went away.
To find this problem, I set breakpoints in Chrome. To do this I followed the online instructions for building a debug version of Chrome and ran it (the debug version of Chrome) under Visual Studio Community edition (2019). The actual debugging was pretty trivial. Building a debug version of Chrome was a substantial effort (almost 40,000 files to compile).
joakime
commented
5 years ago @sbordet @gregw do you think we (jetty) could produce a warning if certain headers are used in the response while on an HTTP/2 connection? Make troubleshooting easier for others in the future?
sbordet
commented
5 years ago @joakime I think that's definitely something that we should do. I have opened #3956.
pschaeffer
commented
5 years ago Mr. Erdfelt,
“do you think we (jetty) could produce a warning”
Not a bad idea. It turns out that HTTP2 disallows quite a few headers (not just Transfer-Encoding). Connection is not allowed for example. Take a look at the links below.
https://www.rhymewithgravy.com/2016/08/23/Beware-the-Connection-Header-in-HTT-P.html
https://github.com/dotnet/corefx/issues/31305
https://http2.github.io/http2-spec/ (see section 8.1.2.2)
https://github.com/nghttp2/nghttp2/issues/887
Thank you
Peter Schaeffer
From: Joakim Erdfelt [mailto:notifications@github.com] Sent: Sunday, August 11, 2019 8:21 AM To: eclipse/jetty.project Cc: Peter Schaeffer; State change Subject: Re: [eclipse/jetty.project] Chrome / Jetty Problems (#3904)
@sbordet https://github.com/sbordet @gregw https://github.com/gregw do you think we (jetty) could produce a warning if certain headers are used in the response while on an HTTP/2 connection? Make troubleshooting easier for others in the future?
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/eclipse/jetty.project/issues/3904?email_source=notifications&email_token=AA4LSSWV2EDISJU7G6X6VZTQEAG37A5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4BAZRQ#issuecomment-520228038 , or mute the thread https://github.com/notifications/unsubscribe-auth/AA4LSSRGNTAROC2DNOYBTZLQEAG37ANCNFSM4IGUYAXA . https://github.com/notifications/beacon/AA4LSSVP4JSZQO3YLHHJ3BLQEAG37A5CNFSM4IGUYAXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4BAZRQ.gif
Dear Sir,
I am having a problem using Chrome with Jetty. At the outset, I must say that I am not sure (at all) if this is a Chrome problem, a Jetty problem, or some error on my part. The symptom is easy to describe. Some HTTP(2) GET requests fail with a -337 error. This is described as a SPDY_PROTOCOL_ERROR in some places. The correct equate is probably HTTP2_PROTOCOL_ERROR. A few notes.
Of course, I can create any other logs or traces that might help resolve this problem. I can also run Jetty under Eclipse and set breakpoints as need be.
Thank you
Peter Schaeffer SPDY Error chrome-net-export-log.zip SPDY Error Wireshark Packets.zip