search

magro / memcached-session-manager

A tomcat session manager that backups sessions in memcached and pulls them from there if asked for unknown sessions
Apache License 2.0
758 stars 348 forks source link

Heap space issue : net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl #410

Open shobero opened 5 years ago

shobero commented 5 years ago

We are using memcache for session storage but on few nodes which are not restarted for more than a week it has been noticed that net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl class has occupied close to or more than 50% of total heap memory and not getting cleared even after major GC run.

Attaching heap memory snapshot for reference. Note : Nodes with this issue has writeQ (java.util.concurrent.LinkedBlockingQueue) size more than 400K (looks like cyclic dependency) where as other nodes which are working fine has writeQ size of 0.

Screen Shot 2019-08-29 at 6 10 52 PM

Detailed snapshot of writeQ

Screen Shot 2019-08-29 at 6 10 37 PM

Can someone help me in figuring out what is the cause of this issue?

mako-fdc commented 5 years ago

@shobero If you use memcached as a backend, this problem might be related to https://github.com/couchbase/spymemcached/pull/38

You might want to check to your tomcat logs for errors, i.e. the catalina.out.

pjain1 commented 3 years ago

We have seen this issue in our cluster as well and this queue which is the operation queue was occupying almost 90% of the heap, I think this happens because of high concurrent load. A workaround is to limit the queue size by implementing custom operation queue like here and using that in the ConnectionFactory. Note - I did not implement any of it, just using it.