We've encountered a memory leak when handling redirects. It looks like when a redirect is encountered by the system, it ends up reusing the existing NettyResponseFuture to handle the request to the new destination. This ends in turn replaces the old request timeout with a new one, orphaning the old timeout. Since references to timeouts are kept in the HashedWheelTimer, and since timeouts have references to the associated RequestTimoutTimerTask, which has a reference to the NettyResponseFuture, none of these objects get cleaned up until the timeout expires and the HashedWheelTimer goes through its periodic cleanup.
In our case, we had accidentially set the request timeout up to 10 hours. We also are using an AsyncResponseHandler to process requests. The AsyncResponseHandler maintains a reference to the ResponseBuilder, and since the Future maintains a reference the AsyncResponseHandler, the result was that all responses from redirected requests were kept in memory until the timeout expired, even though the response handler had finished processing them. Due to oddities of the services we interact with, around 50% of our requests were redirects, causing the system to build up tens of gigabytes of requests in memory in only a couple of hours before the system finally fell over.
slandelle
commented
4 years ago
We've encountered a memory leak when handling redirects. It looks like when a redirect is encountered by the system, it ends up reusing the existing NettyResponseFuture to handle the request to the new destination. This ends in turn replaces the old request timeout with a new one, orphaning the old timeout. Since references to timeouts are kept in the HashedWheelTimer, and since timeouts have references to the associated RequestTimoutTimerTask, which has a reference to the NettyResponseFuture, none of these objects get cleaned up until the timeout expires and the HashedWheelTimer goes through its periodic cleanup.
In our case, we had accidentially set the request timeout up to 10 hours. We also are using an AsyncResponseHandler to process requests. The AsyncResponseHandler maintains a reference to the ResponseBuilder, and since the Future maintains a reference the AsyncResponseHandler, the result was that all responses from redirected requests were kept in memory until the timeout expired, even though the response handler had finished processing them. Due to oddities of the services we interact with, around 50% of our requests were redirects, causing the system to build up tens of gigabytes of requests in memory in only a couple of hours before the system finally fell over.