Memory leaks in Mobicents Sip stack

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Hit the sip b2bua application with huge number of SIP registers and invites 

What is the expected output? What do you see instead?
I would expect the GC on heaps to collect all the expired/invalid sip sessions 
but that doesn't seem to happen and the heap just keeps growing hitting the 
maximum configured values resulting in OutOfMemory Issues

What version of the product are you using? On what operating system?
Product: mss-2.0.0.FINAL-jboss-as-7.1.2.Final-1349104459.zip 
OS: Linux

Please provide any additional information below.
I have a SIP B2BUA application which was deployed on IBM Websphere for a while 
now and now that I want to evaluate JBoss mobicents, I have deployed the same 
application here and the same traffic (SIP registers and invites) have been 
hitting the application. Since it is deployed on Jboss mobicents, I am seeing 
the outofmemory issues periodically which weren't seen as long as the 
application was on IBM Websphere.

Not able to attach the hprof file(heapdump) as it exceeds 10 MB.

Original issue reported on code.google.com by balaraju...@gmail.com on 18 Jul 2014 at 2:39

[GoogleCodeExporter]

Do you invalidate the REGISTER sessions ?
Please move to the latest 
https://mobicents.ci.cloudbees.com/job/Mobicents-SipServlets-Release/lastSuccess
fulBuild/artifact/ and see if it still happens.

Make the heap dump available on a website such as rapidshare or mega. Which 
states are the sessions in ? Who is holding the ref to those sessions ?

Original comment by jean.der...@telestax.com on 18 Jul 2014 at 8:45

[GoogleCodeExporter]

Sorry for the delayed response as I have been trying to upload the heapdump 
file of size 700 MB after compressing it, since couple of days but nothing is 
working out.

We don't invalidate the REGISTER sessions explicitly assuming the stack takes 
care of that and also we haven't changed much on the application for moving to 
mobicents from IBM. I see most of the sessions to be in expired state.

Let me know if the attached references file created from jprofiler, helps.

Original comment by balaraju...@gmail.com on 22 Jul 2014 at 4:04

Attachments:

• Heap_walker_Outgoing_References_2.html

[GoogleCodeExporter]

Hi, 

Was the application on IBM a JSR 116 application ?
The snapshot is not helpful, can you check in which state is the SipSessionImpl 
for 
REGISTER requests ie INITIAL or TERMINATED  state ?

Migration from one vendor to another is not as straigthforward as you would 
think and there can be a lot of differences between container implementations 
or interpretation of the spec and also potential switch of JDK (IBM vs Oracle 
by example). 
Profiling, Fine Tuning and mirgation can be a time consuming task, it may be 
interesting to get helped with some consulting from TeleStax 
http://www.telestax.com/contactus/#InquiryForm

Original comment by jean.der...@telestax.com on 22 Jul 2014 at 9:20

[GoogleCodeExporter]

It's a JSR 289 application. I was still trying to share the heapdumps but no 
luck with any of the online sharing options. Please let me know the way to 
check the state of SipSessionImpl as I don't think I understood your 
requirement here.

Original comment by balaraju...@gmail.com on 24 Jul 2014 at 3:07

[GoogleCodeExporter]

What JVM Options do you use ?
REGISTER sessions are not invalidated automatically upon receiving a 200 OK 
IIRC. I'll need to check if it was on purpose by the spec so you can receive 
multiple REGISTER from the same CSeq-CallID space on the same session or if 
it's a bug. 
Check the sipsession objects in your heap to see if REGISTER sessions are set 
to INITIAL state or TERMINATED state. You can try to invalidate in the 
application the REGISTER sessions also to see if it helps in term of memory 
usage.

Original comment by jean.der...@telestax.com on 24 Jul 2014 at 8:56

[GoogleCodeExporter]

That might be possible as I am not invalidating any sessions explicitly in my 
application and yes, there is a very huge traffic of REGISTERs than INVITEs 
that are hitting my application.

Finally I am able to upload some heap file for you at: 
https://www.dropbox.com/s/5avuae7jw14jz6r/sip_b2bua.zip

JVM options being used for GC are:
-XX:PermSize=1500m -XX:MaxPermSize=2500m -Xms6G -Xmx6G -server 
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSIncrementalDutyCycle=100 
-XX:CMSIncrementalDutyCycleMin=100 -XX:+UseStringCache 
-XX:+OptimizeStringConcat -XX:+UseCompressedStrings -XX:MaxTenuringThreshold=0 
-XX:SurvivorRatio=128 -XX:+UseParNewGC -XX:+UseCompressedOops 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+CMSParallelRemarkEnabled -Xmn1500m

Original comment by balaraju...@gmail.com on 24 Jul 2014 at 4:04

[GoogleCodeExporter]

Have some updates on this? Please help.

Original comment by balaraju...@gmail.com on 28 Jul 2014 at 8:39

[GoogleCodeExporter]

Hi,

This is best effort community support, so we investigate as time permits. In 
the meanwhile, try invalidating your REGISTER Sessions to see if that helps.

If you need better response time, you can purchase support services from 
http://www.telestax.com/contactus/#InquiryForm or investigate the problem from 
source code and memory heap investigation and propose a patch 

Original comment by jean.der...@telestax.com on 29 Jul 2014 at 9:00

[GoogleCodeExporter]

Thanks for your reply, Jean. I will continue working on invalidating the 
REGISTER sessions to see if that helps.

I have already communicated my company's legal team on the requirement of 
support licenses and hopefully they have started working on that. Meanwhile I 
am trying to prove that this is the best fit for our application by fixing most 
of the issues that came up.

Original comment by balaraju...@gmail.com on 29 Jul 2014 at 2:20

[GoogleCodeExporter]

Hi Jean,

I have enabled the mobicents logs for my application and got to see that the 
transaction is already being removed from client transaction table upon 
receiving 200 OK for REGISTERs. I suppose there wouldn't be any session post 
this, to invalidate it. Please correct me. Thanks !  

Trace attached for a SIPp REGISTER fired onto my application (for reference).

Original comment by balaraju...@gmail.com on 4 Aug 2014 at 7:27

Attachments:

• Mobicents_Trace.log

[GoogleCodeExporter]

Issue was moved to GitHub:

https://github.com/Mobicents/sip-servlets/issues/36

Original comment by desi.pep...@telestax.com on 18 Dec 2014 at 6:57

[GoogleCodeExporter]

Original comment by jean.deruelle on 18 Dec 2014 at 9:27

• Changed state: Moved

[GoogleCodeExporter]

REGISTER sessions needs to be invalidated by the application otherwise they 
stay in INITIAL state.

JSR 289 States in Section 6.2.1 Relationship to SIP Dialogs

"
...
2. In general, whenever a non-dialog creating request is sent or received, the 
SipSession state remains unchanged. Similarly, a response received for a 
non-dialog creating request also leaves the SipSession state unchanged. The 
exception to the general rule is that it does not apply to requests (e.g. BYE, 
CANCEL) that are dialog terminating according to the appropriate RFC rules 
relating to the kind of dialog.
...
"

The rationale for that is explained in the same section :
"
...
The INITIAL state is introduced to allow a UAC to generate multiple requests 
with the same Call-ID, From (including tag), and To (excluding tag), and within 
the same CSeq space. This is useful, for example, in the following situations:
...
* REGISTER requests sent from a UAC to the same registrar should all use the 
same
Call-ID header field value and should increment the CSeq by one for each 
request sent [RFC 3261, section 10.2].
...
"

Original comment by jean.deruelle on 18 Mar 2015 at 1:33