search

twitter / finagle

A fault tolerant, protocol-agnostic RPC system
https://twitter.github.io/finagle
Apache License 2.0
8.78k stars 1.45k forks source link

StdStackServer connections ConcurrentHashmap holding on to connections and running out of memory #872

Open vikramkadi opened 4 years ago

vikramkadi commented 4 years ago

Finagle http server(version: 6.39.0) seems to not release older connections from the concurrenthashmap/set increasing old gen space overtime.

Expected behavior

The connections bookkeeping set/map should clean up older entries(https://github.com/twitter/finagle/blob/finagle-6.39.0/finagle-core/src/main/scala/com/twitter/finagle/server/StackServer.scala#L230).

Actual behavior

On taking a jvm heap dump at multiple times during the day for a high qps finagle server, we see that the connections set/map is not flushing out some of the older connections that just stay and keep taking up large amounts of memory

Steps to reproduce the behavior

enbnt commented 4 years ago

Hi @vikramkadi, have you tried reproducing this issue with a more recent version of Finagle? The 6.39 version appears to be from 2016 and the code has changed significantly. I would recommend trying an upgrade and seeing if the issue persists. If it is still there on the latest release, we will need to gather some more context to assist. Thank you.