sessions lost on Tomcat 7 reload

[GoogleCodeExporter]

What steps will reproduce the problem?

1. Tomcat 7.0.34
2. The following jar files are located in $CATALINA/lib
   memcached-session-manager-1.6.4.jar
   memcached-session-manager-tc7-1.6.4.jar
   msm-kryo-serializer-1.6.4.jar
   kryo-1.0.4.jar
   kryo-serializers-0.10.jar
   asm-3.2.jar
3. $CATALINA/conf/context.xml configured with
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:[server1]:11211,n2:[server2]:11211"
    failoverNodes="n2"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$" transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"/>

I would like to be able to replace a JAR file for our app and reload the app 
through the Tomcat manager without losing sessions.  However, when I reload the 
application all the sessions are lost and I'll see a 'ClassNotFoundException' 
related to one of the classes in our app (which exists in both the old jar file 
and the new jar file)

Note: I put all the various memcache jars and dependencies in $CATALINA/lib 
hoping to avoid having them in each webapp (there are a number of webapps and 
as this is a dedicated server environment I was hoping to avoid duplication of 
common jars)

So the question is:  do I have a misconfiguration somewhere?   How can I safely 
reload an app so it picks up new application code without losing sessions?

Please provide any additional information below.

Original issue reported on code.google.com by st...@regattacentral.com on 6 Aug 2013 at 1:29

[GoogleCodeExporter]

Does "normal" failover work for you, so that tomcat2 picks up sessions from 
tomcat1 when this is stopped?
Does it work when you reload tomcat without changing any jar?

Regarding jars in $CATALINA/lib: this should work as long as the session 
contains only objects of standard classes. As soon as it contains classes from 
your webapp deserialization should fail (as the classes are not available in 
the common classloader, for details of tomcat classloaders you might read 
http://tomcat.apache.org/tomcat-7.0-doc/class-loader-howto.html).

Original comment by martin.grotzke on 6 Aug 2013 at 9:13

[GoogleCodeExporter]

Martin,

Currently I'm running Tomcat on just one of the two servers (memcached is 
running on both and sessions appears to be getting replicated fine).  

The webapp does indeed contain custom classes which are session-scoped.  So it 
sounds like maybe this is a classloader issue related to having the memcached 
dependendant jar files in $CATALINA/lib instead of the WEBINF/lib?  I was 
hoping to avoid duplicating those jar files in each of the webapps but that is 
easy enough to do if that fixes this.

Original comment by st...@regattacentral.com on 6 Aug 2013 at 5:53

[GoogleCodeExporter]

Can you try placing jars according to the documentation? 

Original comment by martin.grotzke on 6 Aug 2013 at 7:49

[GoogleCodeExporter]

Martin,

I moved the following jar file from ~CATALINA/lib to ROOT/WEB-INF/lib 
1. asm-3.2.jar
2. kryo-1.0.4.jar
3. kryo-serializers-0.1.0.jar
4. minlog-1.2.jar
5. msm-kryo-seralizer-1.6.4.jar
6. reflectiasm-1.01.jar 

When I restarted Tomcat the ROOT webapp came up fine but the Tomcat manager app 
no longer worked.   

Note: I setup the memcaching to work globally via ~CATALINA/context.xml similar 
to the example in the docs.

Logs show:

 Caused by: java.lang.ClassNotFoundException: de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory

This makes sense kinda given I setup the memcached serialization globally per 
the example in the doc but the manager app wouldn't have access to the six jar 
files.

I put the jar files back into ~CATALINA/lib (as well as the WEBINF/lib of my 
ROOT webapp and everything is running. But when I simply reload the webapp (no 
changes) my session count goes to 0.  So I'm assuming it is still not setup 
correctly.

Thank you in advance for suggestions.  I look forward to getting this working 
correctly,

Steve

Original comment by st...@regattacentral.com on 7 Aug 2013 at 7:27

[GoogleCodeExporter]

Not sure which session count you're referring to, but after reloading tomcat, 
the manager app cannot list/know sessions because they're loaded from memcached 
when requested.

Does tomcat failover work for your? E.g. when you restart tomcat, do you get 
back your session? I'd like to see if there's a difference between restart and 
reload in your case.

Btw, some debug log output would be helpful to analyse this, for configuration 
please check 
https://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#C
onfigure_logging.

To get the tomcat manager app working (with mentioned jars in WEB-INF/lib) you 
should define msm in the webapp specific context xml (e.g. in 
META-INF/context.xml or in $CATALINA_BASE/conf/[enginename]/[hostname]/ROOT.xml 
if this is already existing). Please check the tomcat docs 
http://tomcat.apache.org/tomcat-7.0-doc/config/context.html#Defining_a_context 
for details.

Original comment by martin.grotzke on 7 Aug 2013 at 8:00

[GoogleCodeExporter]

Martin,

Thank you again for your assistance.  Regretfully, I am not getting the 
solution I'm hoping for.   So perhaps I should first confirm my assumptions 
with you...

I would like to be able to upload an update to a jar file to WEB-INF/lib and 
then do a 'reload' in Tomcat Manager so it picks up the newer version of the 
jar file while maintaining sessions.   So this isn't so much about failover to 
another memcached server so much as maintaining sessions across reloads.

I enabled logging and when I reload the ROOT webapp via the Tomcat Manager I 
see the following in the logs:

Aug 21, 2013 8:21:39 AM de.javakaffee.web.msm.MemcachedBackupSessionManager 
stopInternal
INFO: Removing sessions from local session map.
Aug 21, 2013 8:21:39 AM de.javakaffee.web.msm.MemcachedSessionService shutdown
INFO: Stopping services.

That second line about removing session from local session map seem to confirm 
what I see in actual tests - my sessions get lost and logged in users abruptly 
lose their session.

Note: at the moment I have one Tomcat running and two instances of memcached.  
is it possible with memcache to maintain the sessions during reloads with just 
one Tomcat running?

Also, I moved app-specific jars out of ~CATALINA/lib and into the WEBINF/lib 
and also upgraded to latest versions as I saw a changelog reference to 
supporting redeploys (and possibly reloads?) in 1.6.5.

/CATALINA/lib/memcached-session-manager-1.6.5.jar
/CATALINA/lib/memcached-session-manager-tc7.1.6.5.jar
/CATALINA/lib/spymemcached-2.9.1.jar

~/ROOT/WEB-INF/lib/kryo-1.0.4.jar
~/ROOT/WEB-INF/lib/kryo-serializers-0.10.jar
~/ROOT/WEB-INF/lib/msm-kryo-serializer-1.6.4.jar
~/ROOT/WEB-INF/lib/minlog-1.2.jar

And in ~/ROOT/META-INF/context.xml...
<Context>
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:server1:11211,n2:server2:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
    transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
    />
</Context>

Thanks,
Steve

Original comment by st...@regattacentral.com on 21 Aug 2013 at 3:36

[GoogleCodeExporter]

I just noticed something very peculiar..   When I launch Tomcat and start a few 
sessions I see the expected n1 or n2 suffix on the session identifiers.  But 
when I reload the webapp via the Tomcat Manager not only are those sessions 
lost but any new sessions no longer have the n1/n2 suffix.  Its as if the 
reloaded webapp is no longer using memcached session manager.

Original comment by st...@regattacentral.com on 21 Aug 2013 at 4:07

[GoogleCodeExporter]

I've never tested reloading tomcat via the manager app. I've tested restarting 
tomcat to see if sessions are pulled in after restart as expected.

Does this work for you (when you restart tomcat, do you get back your session)? 
I'd like to see if there's a difference between restart and reload in your case.

Original comment by martin.grotzke on 21 Aug 2013 at 9:15

[GoogleCodeExporter]

restart appears to work ok.  Would be preferential though to support reloads as 
it would allow us to take only a specific app offline for a moment instead of 
the whole server.  

i did notice a possible inconsistency in the documentation (or perhaps I'm 
confused). 

The docs at 
http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration 
say to, "Update the <Context> element (in $CATALINA_HOME/conf/context.xml" but 
to put the serialization jars in the ROOT webapp's WEB-INF/lib.    When I did 
this I found that the Tomcat Manager app fails to startup, which makes sense I 
think because it doesn't have the kryo serializer jars in its classpath.  To 
get around this my originally had been to put the kryo jars into ~CATALINA/lib 
just as say JDBC jars.  You may recall from the beginning of this thread that 
this didn't work because of classloader issues.

Any recommendations on that?  

Also, would you like me to create a new issue to request support for sessions 
maintained between reloads or will this issue (#174) suffice?

Thanks as always,
Steve

Original comment by st...@regattacentral.com on 21 Aug 2013 at 9:27

[GoogleCodeExporter]

> restart appears to work ok.  Would be preferential though to support reloads 
as it would allow us to take only a specific app offline for a moment instead 
of the whole server. 

So I have to debug what's happening during reload, that's what this issue is 
for (no need to create a new one). Not sure when I will find time for this, if 
you want to dig deeper into this it would be great.

Regarding docs inconsistencies: thanx for pointing this out, I just changed it 
to META-INF/context.xml.

Original comment by martin.grotzke on 21 Aug 2013 at 9:37

• Changed state: Accepted

[GoogleCodeExporter]

I'm not sure I'd be much help looking at the internals to figure this one out.  
However, I'm happy to make a donation for all of your work in lieu of me trying 
to fix it. Do you accept donations?

Original comment by st...@regattacentral.com on 23 Aug 2013 at 6:21

[GoogleCodeExporter]

What a huge offer, thanx! Yes, I'm accepting donations, although I cannot 
promise that I can fix it (not yet sure what's the reason for the issue) and 
when I'll have a solution. I'll have a look at this issue and let you know 
about the progress. But I don't make this dependant on a donation from you, so 
feel free in this regard! :-)

Original comment by martin.grotzke on 23 Aug 2013 at 7:59

[GoogleCodeExporter]

completely understood. donation is more as a gesture as I'm not able to 
contribute with actual code and as a thank you for what you have already given 
the community.   I don't see a link to donate. can you send that?

Original comment by st...@regattacentral.com on 23 Aug 2013 at 8:30

[GoogleCodeExporter]

Hmm, I haven't thought about a donate button yet :-) Perhaps you could use 
paypal and send to my email mg<at>fsfe<dot>org.

Original comment by martin.grotzke on 24 Aug 2013 at 8:47

[GoogleCodeExporter]

Fixed, the msm stop() implementation was not cleaning up internal state 
completely.
Thanx for reporting this!

I'm attaching the jar with the fix (it's compatible with the tc7 jar), or do 
you prefer a released version?

Original comment by martin.grotzke on 24 Aug 2013 at 10:33

• Changed state: Fixed
• Added labels: Milestone-1.6.6

Attachments:

• memcached-session-manager-1.6.6-SNAPSHOT.jar

[GoogleCodeExporter]

Btw, thanx for your donation, Steve!

Original comment by martin.grotzke on 27 Aug 2013 at 7:34

[GoogleCodeExporter]

Original comment by martin.grotzke on 20 Dec 2013 at 10:22

• Added labels: Milestone-1.7.0
• Removed labels: Milestone-1.6.6