search

magro / memcached-session-manager

A tomcat session manager that backups sessions in memcached and pulls them from there if asked for unknown sessions
Apache License 2.0
759 stars 348 forks source link

Reconnecting to memcache fails on run time #374

Open desaivishal42 opened 6 years ago

desaivishal42 commented 6 years ago

de.javakaffee.web.msm.MemcachedSessionService loadFromMemcached WARNING: Could not load session with id 95A5E60EE19533ECBF1888D8526CECE4 from memcached. xxxxxxxxx/xx.x.xxx.xxx:11211 active: true, authed: true, last read: 7,103 ms ago } at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1240) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at de.javakaffee.web.msm.MemcachedSessionService.loadFromMemcached(MemcachedSessionService.java:1092) at de.javakaffee.web.msm.MemcachedSessionService.findSession(MemcachedSessionService.java:585) at de.javakaffee.web.msm.MemcachedBackupSessionManager.findSession(MemcachedBackupSessionManager.java:216) at org.apache.catalina.connector.Request.doGetSession(Request.java:2981) at org.apache.catalina.connector.Request.getSession(Request.java:2378) at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:897)

--

INFO net.spy.memcached.MemcachedConnection: Reconnecting due to exception on {QA sa=xxxxxxxxx/xx.x.xxx.xxx:11211, #Rops=6, #Wops=0, #iq=0, topRop=Cmd: get Keys: 95A5E60EE19533ECBF1888D8526CECE4Exp: 0, topWop=null, toWrite=0, interested=1} java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384) at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:784) at net.spy.memcached.MemcachedConnection.handleReadsAndWrites(MemcachedConnection.java:684) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:647) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:418) at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:1401)

WARN net.spy.memcached.MemcachedConnection: Closing, and reopening {QA sa=xxxxxxxxx/xx.x.xxx.xxx:11211, #Rops=6, #Wops=0, #iq=0, topRop=Cmd: get Keys: 95A5E60EE19533ECBF1888D8526CECE4Exp: 0, topWop=null, toWrite=0, interested=1}, attempt 0. net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding partially completed op: Cmd: get Keys: 95A5E60EE19533ECBF1888D8526CECE4Exp: 0

WARN net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding partially completed op: Cmd: get Keys: 95A5E60EE19533ECBF1888D8526CECE4Exp: 0

WARNING: Could not load session with id 95A5E60EE19533ECBF1888D8526CECE4 from memcached. java.util.concurrent.CancellationException: Cancelled at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177) at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at de.javakaffee.web.msm.MemcachedSessionService.loadFromMemcached(MemcachedSessionService.java:1092) at de.javakaffee.web.msm.MemcachedSessionService.findSession(MemcachedSessionService.java:585) at de.javakaffee.web.msm.MemcachedBackupSessionManager.findSession(MemcachedBackupSessionManager.java:216) at org.apache.catalina.connector.Request.doGetSession(Request.java:2981) at org.apache.catalina.connector.Request.getSession(Request.java:2378)

WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for 95A5E60EE19533ECBF1888D8526CECE4. WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for 95A5E60EE19533ECBF1888D8526CECE4. WARN net.spy.memcached.MemcachedConnection: Could not redistribute to another node, retrying primary node for 95A5E60EE19533ECBF1888D8526CECE4. de.javakaffee.web.msm.MemcachedSessionService loadFromMemcached WARNING: Could not load session with id 95A5E60EE19533ECBF1888D8526CECE4 from memcached. net.spy.memcached.OperationTimeoutException: Timeout waiting for value: waited 1,000 ms. Node status: Connection Status { xxxxxxxxx/xx.x.xxx.xxx:11211 active: false, authed: true, last read: 14,666 ms ago } at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1240) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at de.javakaffee.web.msm.MemcachedSessionService.loadFromMemcached(MemcachedSessionService.java:1092)

I am storing tomcat session in memcache and using spymemcache library for that. There was an issue on running server where it could not found session from memcache --> Got some exception --> tried to reconnect to memcache automatically --> when started the memcache flag - active was false.

What could be the reason for this issue ?

Below are the configurations on context file

WEB-INF/web.xml
twu027 commented 6 years ago

I got the similar error, but in my case, I got the port number wrong. 11211 is the default port which memcached server is running at. So seems like you got it right. Are you able to ping your server name or ip address manually? The error is obvisous, the net.spy.memcached.MemcachedConnection is unable to make connections.

desaivishal42 commented 6 years ago

Hey twu027, Thanks for reply. This issue occurred on live environment and as you can see there was an exception and then it automatically tried to close and open the connection, it could not able to connect to memcache later after the connection was closed. Also if you can see it returned "active = false" when tried to reconnect again.

When I have restarted the tomcat server, it re-established the connection and started working fine again. So I am confused, is there any configuration which is not allowing to connect it again when connection is closed automatically ?

twu027 commented 6 years ago

Not sure about that. So it is working in your environment now after you restart tomcat server? Do you have to restart tomcat server regularly to make this work? There are people using this library but seems like there isn't much support from the creator anymore.

I couldn't even get it working at all. My connection issue is gone after I set the port number correctly, not my issue is some of the classes in my app doesn't implement the serializable interface so I can't even serialize the session attributes which is bizarre.

magro commented 6 years ago

As a wild guess, maybe the ip address had changed (in case a host name was configured), so that inetaddress dns caching caused the issue?

Is this issue still relevant at all, or shall we close it?