Closed andrewxian closed 1 year ago
lachlan-roberts
commented
2 years ago @andrewxian Version 9.4.6 is very old, please upgrade to the latest version Jetty 9 (currently 9.4.44.v20210927). There has been many bug fixes and security fixes since 9.4.6 so it is likely the issue you are experiencing is fixed in the latest version.
andrewxian
commented
2 years ago Hi @lachlan-roberts,
Unfortunately, it is not easy for us to upgrade the jetty version as the issue happened on customer's site.
I agree with you that upgrading the jetty version is the best option, but it takes some procedures.
In the meantime, can I ask what jetty is trying to do in the stack trace? I would be very grateful if that could be explained to me.
Thanks.
joakime
commented
2 years ago In the meantime, can I ask what jetty is trying to do in the stack trace? I would be very grateful if that could be explained to me.
It appears to be reading/parsing from an SSL/TLS connection. Without a DEBUG log, the exact reason for this read/parse is unknown, the stacktrace just tells us a generic detail at that 1 point in time.
Also, an important note, since you are using SSL/TLS you must keep your JVM up to date.
Java 1.8.0_144 expired on October 17, 2017, and contains none of the updated SSL/TLS cryptograph databases and updated security configurations that being on the public facing internet requires when using SSL/TLS. See:
The reasons for your high CPU usage can easily be explained by the out of date cryptographic configuration with your JVM version. (Your JVM is working way harder to satisfy the requirements from everyone else that has upgraded HTTP clients or client environments. Example: your JVM is likely falling back to highly inefficient crypto providers as the better ones are simply not available on your JVM).
Since Java 1.8.0_144 entire SSL/TLS protocol levels have been whittled down and are now completely disabled in the JVM. Your configuration in 1.8.0_144 is almost guaranteed to be considered unsafe as you don't even have the newer protocols (and cipher suites) present in that old version of Java. Go ahead, test your site and see - https://www.ssllabs.com/ssltest/
The primary reasons for keeping your JVM up to date are (in order of priority)
dimas
commented
2 years ago Given your CPU remains high even without any traffic, it could be something similar to the issue we had even though stacktrace is different - https://github.com/eclipse/jetty.project/issues/6973 In our case issue was triggered by problems in logback library but I believe the root cause is somewhere in Jetty code where some edge case allowed reading from already closed connection or something like that.
If you affected by something similar, I doubt you can just tune something to improve things. My suspicion is that in our case problem was triggered by some some guys running vulnerability scanners against us so we were seeing some invalid requests that kicked things off.
github-actions[bot]
commented
1 year ago This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.
github-actions[bot]
commented
1 year ago This issue has been closed due to it having no activity.
Jetty version(s) 9.4.6
Java version/vendor
(use: java -version)java -version java version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)OS type/version Windows 10
Description We had an issue of CPU spike usage on the jetty server with little or no traffic on an idle node. Restarting the app (with Jetty) would fix the problem.
I used some top tools and saw several QTP threads had consumed most of the CPU usages with similar stack trace:
I have some questions for the above two stack traces:
How to reproduce? Unfortunately, I had no idea how to reproduce it. The only thing worth mention is that issue happened every Saturday.