search

jetty / jetty.project

Eclipse Jetty® - Web Container & Clients - supports HTTP/2, HTTP/1.1, HTTP/1.0, websocket, servlets, and more
https://eclipse.dev/jetty
Other
3.85k stars 1.91k forks source link

Memory leak via IdleTimeout and supressedExceptions in Jetty 12.0.1 #10564

Open jaroslawr opened 1 year ago

jaroslawr commented 1 year ago

One of clients connecting to our HTTP server cluster is causing GZIPContentDecoder errors apparently by sending invalid gzip, at a request rate that is very low compared to the overall handled request rate, which is OK. However I noticed that the resulting exception carries with itself in the surpressedExceptions field an ArrayList of tens of thousands of TimeoutExceptions from IdleTimeout. I attach some screenshots from VisualVM heap analysis that hopefully make the issue clear.

First screen shows the gzip related exception along with the long chain of supressedExceptions:

exceptionLeak1

Second screen shows one of the surpressedExceptions with its stack trace (for whatever reason stackTrace field is empty for those instances so the stack trace is more "raw" than the previous one, from the backtrace field):

exceptionLeak2

These TimeoutExceptions seem to somehow accumulate over time on the heap.

joakime commented 1 year ago

You sure it's 12.0.1?

This seems a lot like Issue #10120 reported against 12.0.0 betas, and fixed with the final 12.0.0 release.

jaroslawr commented 1 year ago

Yes I am sure, we never used the beta releases.

lorban commented 1 year ago

@jaroslawr Sorry for the delay getting back to you.

Have you got any news about this issue? I checked the GZIPContentDecoder failure path, and there is no obvious way suppressed exceptions could accumulate over time. The RuntimeException thrown when a decoding error occurs is forwarded via fail() to the Content.Source from which the bytes were read, which eventually should be the HttpChannelState, the latter eventually clearing its exception in recycle() once the request has been fully processed.

You VisualVM screenshot does not show which object references that exception with thousands of suppressed exceptions, so which one was it? Is it changing over time?

Any more information you can collect would help track down the problem.

github-actions[bot] commented 6 days ago

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.