Running duration test in AWS using memecache session failover jars version 1.6.5 and getting 403 forbitten response after few minutes into the test.

[GoogleCodeExporter]

we are using the following settings in the context.xml
<Context>
    <!-- Default set of monitored resources -->
    <WatchedResource>WEB-INF/web.xml</WatchedResource>
    <!-- Uncomment this to disable session persistence across Tomcat restarts -->
    <!--
    <Manager pathname="" />
    -->
    <!-- Uncomment this to enable Comet connection tacking (provides events
         on session expiration as well as webapp lifecycle) -->
    <!--
    <Valve className="org.apache.catalina.valves.CometConnectionManagerValve" />
    -->
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
     memcachedNodes="performance2.puh5jb.cfg.use1.cache.amazonaws.com:11211"
     requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
     transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
     sessionBackupAsync="false"
     sessionBackupTimeout="100"
     sticky="true"
     copyCollectionsForSerialization="false"
     />
</Context>

Original issue reported on code.google.com by gagan.jo...@snapwiz.com on 11 Dec 2013 at 1:26

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

We have sticky session in the AWS with 300 ms in elastic beanstalk to 
distribute the load on the APP servers. Its a clustor where we have 3 memcache 
nodes and session failover jars 1.6.5 are also configured. In the Initial stage 
the test goes well but after 25 minutes(approx), the test starts getting 403 
errors. 
we have tried various changes like Support context configured with 
cookies="false" with this change we are manually able to reproduce with the 
single user. The user logs in the application and navigation to any page throws 
403 forbitten error. Can you please let me know what is the solution and if we 
need to modify any paramters in our context.xml in tomcat7.

Original comment by gagan.jo...@snapwiz.com on 11 Dec 2013 at 1:34

[GoogleCodeExporter]

Hi Martin,

Can you please update if anyone is working on the issue, i have added more logs 
and i am surprised to see the memcachedExpirationTime: 0 for few of the 
sessions and its the same time I started seeing the 403 error in the response 
as well. Is there any way to set the session timeout in Memcache as well.

I am also suspecting the issue can be similar to the open which some one has 
also reported. Reference 
http://grokbase.com/t/gg/memcached-session-manager/138hefbf0b/about-session-expi
ration#top  

Dec 31, 2013 7:35:58 AM de.javakaffee.web.msm.MemcachedSessionService 
updateExpirationInMemcached
FINE: Checking session 63E794469F2C90E5324FFBBB752CAFCC: 
- isValid: true
- isExpiring: false
- isBackupRunning: false
- isExpirationUpdateRunning: false
- wasAccessedSinceLastBackup: true
- memcachedExpirationTime: 0
Dec 31, 2013 7:35:58 AM de.javakaffee.web.msm.MemcachedSessionService 
updateExpirationInMemcached
FINE: Checking session 2CB2EF52A771D0DF8761029FDAE18E7F: 
- isValid: true
- isExpiring: false
- isBackupRunning: false
- isExpirationUpdateRunning: false
- wasAccessedSinceLastBackup: true
- memcachedExpirationTime: 1397
Dec 31, 2013 7:35:58 AM de.javakaffee.web.msm.MemcachedSessionService 
updateExpirationInMemcached
FINE: Checking session DD546F77E8C64E98F9A2397E65941E77: 
- isValid: true
- isExpiring: false
- isBackupRunning: false
- isExpirationUpdateRunning: false
- wasAccessedSinceLastBackup: false
- memcachedExpirationTime: 1667
Dec 31, 2013 7:35:58 AM de.javakaffee.web.msm.MemcachedSessionService 
updateExpirationInMemcached
FINE: Checking session 9D823BBC062883FC6140D21B11FF888F: 
- isValid: true
- isExpiring: false
- isBackupRunning: false
- isExpirationUpdateRunning: false
- wasAccessedSinceLastBackup: true
- memcachedExpirationTime: 1006
Dec 31, 2013 7:35:58 AM de.javakaffee.web.msm.MemcachedSessionService 
updateExpirationInMemcached
FINE: Checking session 75002597BCAC625624A87AAB80BA2CAE: 
- isValid: true
- isExpiring: false
- isBackupRunning: false
- isExpirationUpdateRunning: false
- wasAccessedSinceLastBackup: true
- memcachedExpirationTime: 0
Dec 31, 2013 7:35:58 AM de.javakaffee.web.msm.MemcachedSessionService 
updateExpirationInMemcached

Please let me know in case you need anymore information to investigate further. 
Thanks in Advance!

Original comment by gagan.jo...@snapwiz.com on 31 Dec 2013 at 7:57

[GoogleCodeExporter]

At first sorry for the long delay! 
No, I'm currently not working on this (these days working on tomcat8 support 
and now pluggable backends / redis support).

Can you provide a step by step guide that allows me to reproduce this issue 
(including all needed information regarding your app), with the smallest 
possible session timeout to reduce the time needed for analyzing this?

Please provide the log output you get when you're following this guide 
yourself, with activated debug logging (check docs for details).

What does "300 ms in elastic beanstalk" refer to, 300 millis of what?

Because you're getting 403 errors it seems you're using authentication - which 
kind of? Does it matter for this issue that you're using authentication?

What's the configured session-timeout in web.xml?

Original comment by martin.grotzke on 1 Jan 2014 at 11:38

[GoogleCodeExporter]

Hi Martin,

Thanks for the reply.

The background info which will help you to investigate the issue.
1. Environment details:
Auto Scaling 2:5
Added session failover jars 1.7.0
Stickiness in AWS : 300 seconds (its not 300ms sorry for the typo mistake)
2. I have conducted the test without session failover jars, I am able to 
reproduce the error after 5 minutes as we have stickiness 300 seconds (5 
minutes) in AWS. Which is correct behavior as the session switching between the 
app servers will not work after 5 minutes.
3. We don't see any errors in the app server logs when we start the server its 
taking the latest jars in the /usr/share/tomcat7/lib location, we also tried to 
copy the jars /usr/share/tomcat7/webapps/ROOT/WEB-INF/lib location as well
asm-3.3.1.jar
reflectasm-1.01.jar
minlog-1.2.jar
kryo1-1.05.jar
msm-kryo-serializer-1.7.0.jar
kryo-serializers-0.11.jar
memcached-session-manager-tc7-1.7.0.jar
memcached-session-manager-1.7.0.jar
couchbase-client-1.1.4.jar
AmazonElastiCacheClusterClient-1.0.jar

Session details in Web.xml is 30 minutes:
 <session-config>
        <session-timeout>30</session-timeout>
    </session-config>

I have conducted lot of test to figure out the root case, please see the 
conclusion below:
1. Increased Stickiness to 3 hours in AWS, I don't see any 403 error as the 
session are sticky.
2. Reduced the Stickiness to 300 seconds, and having the session failover jars 
in the App servers, the test going fine till 30 minutes approx., but after that 
its throwing 403 error.
3. Removing the session failover jars we see the error after 5 minutes as 
mentioned above.
4. I have conducted one test with single app server and I don't see any 403 
error with session failover jars and stickiness 300 seconds, no errors with 
high load like 300 users.
5. I have conducted a lesser load test with 30 users with session failover jars 
and stickiness 300 seconds and 2 app servers , no errors.

These conclude something wrong going in the memcache as after certain duration 
in case of high user load like 1000 users with 2 App servers. Its throwing 403 
error.
Can you please let me know do we have any session active duration in memcache, 
as in application session will expire after 30 minutes in case of Inactive 
users but the test users are active and they are performing various 
transactions. 
I thing this is the problem which other people are also facing in case of load 
testing. Please look into the issue it will be helpful for other people as well.

Please let me know in case you need any more information to investigate.
Thanks in Advance! 

Original comment by gagan.jo...@snapwiz.com on 2 Jan 2014 at 7:53

[GoogleCodeExporter]

Re session expiration in memcached: msm sets the timeout for objects stored in 
memcached to the session-timeout as configured in tomcat. As an optimization 
msm updates the session in memcached only if the session was changed by the 
application. If a session has not been changed by the application, shortly 
before the timeout (in memcached) msm updates the expiration time in memcached.

What does your test do? Is the session modified on each request?

According to your tests it seems that the number of users impact the results. 
Is it possible that with many users the cache capacity is exceeded and 
therefore sessions are dropped?
-> Which node type are you using for your cache nodes / how many memory is 
available?
-> Can you try to set error_on_memory_exhausted to true?
-> Can you try to determine the max number of users / sessions that you can 
store in elasticache?

Original comment by martin.grotzke on 2 Jan 2014 at 12:24

[GoogleCodeExporter]

User login and keep on performing the activities for hours (1-2 hour) and exit. 
No session is not modified on each request.

we are using medium node with 3.75 GB ram. I just checked the Graphs the free 
able memory is 2+ GB approx. Unused memory 3 GB approx.
I set the error_on_memory_exhausted to true. No errors in the application logs.

I have conducted the test with 2000 users load, I still see 403 errors.
I can give to more details like how many users / Sessions the elasticache can 
store in few more days.
Can you please investigate as I don't see the problem with the memory on 
memcache node.
Thanks you!

Original comment by gagan.jo...@snapwiz.com on 2 Jan 2014 at 3:51

[GoogleCodeExporter]

Because you ask me to investigate I repeat things I already asked for:
- Please provide a step by step guide that allows me to reproduce this issue 
(including all needed information regarding your app), with the smallest 
possible session timeout to reduce the time needed for analyzing this. It would 
be great if your step by step guide would target a plain tomcat + memcached 
setup, because I'd be happy if I wouldn't have to spend money for AWS stuff in 
addition to the time I'm investing for msm maintenance/support. (Btw, I'm 
thinking about a commercial support offering that allows people to buy time and 
set priorities for development/support, but I haven't thought about the details 
enough - if you're interested in this we could chat privately).

- Please provide the log output you get when you're following this guide 
yourself, with activated debug logging (check docs for details).

- Because you're getting 403 errors it seems you're using authentication - 
which kind of? Does it matter for this issue that you're using authentication?

Original comment by martin.grotzke on 2 Jan 2014 at 4:22