search

unispeech / unimrcp

Open source cross-platform implementation of MRCP protocol
http://www.unimrcp.org
Apache License 2.0
370 stars 163 forks source link

Memory leak when loading/unloading Asterisk res_speech_unimrcp.so - possibly a bug in Asterisk #53

Closed achaloyan closed 9 years ago

achaloyan commented 9 years ago

Originally reported on Google Code with ID 53

Hi there,

I've used on a CentOS 5.4 (2.6.18-164.6.1.el5PAE, gcc 4.1.2) server:
http://unimrcp.googlecode.com/files/uni-ast-package.tar.gz

and from this .tar.gz I have installed:
 - Asterisk 1.6.10
 - APR 1.3.8
 - APR-Util 1.3.9
 - Sofia SIP 1.2.10
 - UniMRCP 0.8.0

The process I followed, was:
# ./ast-install.sh
# cd asterisk
# make samples
# cd ..
# ./uni-install.sh
# cd asterisk-unimrcp
# ./configure
# make
# make install

I then started Asterisk in a console (asterisk -c) and looked at memory 
usage. Asterisk's memory usage starts at virt=98388,res=14404 and after 
3 "module unload res_speech_unimrcp", "module load res_speech_unimrcp", it 
was at virt=190596,res=14548. After another 3 unload,loads, it was at 
virt=282804,res=14652.

In trying to isolate the leak, I used valgrind. I then started Asterisk in 
valgrind (valgrind --leak-check=full --show-reachable=yes --log-
file=/root/uni-ast-package.valgrind.log /usr/sbin/asterisk -c) and checked 
the memory usage while doing "module unload res_speech_unimrcp", "module 
load res_speech_unimrcp". Memory usage tends to grow continuously without 
me even loading/unloading, so I think that's probably valgrind messing up.

I shut asterisk down using "core stop gracefully" after I did a "module 
unload res_speech_unimrcp", just to make sure (see uni-ast-
package.valgrind.log.9287). If I don't do a "module unload 
res_speech_unimrcp", it shows some leaks in unimrcp, but I'm not sure 
whether Asterisk doesn't unload the module properly - although it did 
print the following (see uni-ast-package.valgrind.log.10535):

*CLI> core stop gracefully
2009-11-30 22:08:27:858884 [INFO]   Terminate MRCP Client
[Nov 30 22:08:27] NOTICE[10535]: res_speech_unimrcp.c:929 unload_module: 
Unload UniMRCP module
2009-11-30 22:08:27:863328 [INFO]   Terminate SofiaSIP Agent
2009-11-30 22:08:27:867263 [INFO]   Terminate UniRTSP Agent
2009-11-30 22:08:27:912252 [INFO]   Receive SIP Event [nua_r_shutdown] 
Status 200 Shutdown successful
2009-11-30 22:08:27:937971 [INFO]   Terminate TCP/MRCPv2 Connection Agent
2009-11-30 22:08:27:943990 [INFO]   Terminate Media Processing Engine
2009-11-30 22:08:27:970040 [NOTICE] MRCP Client Terminated
2009-11-30 22:08:27:977354 [INFO]   Destroy SofiaSIP Agent
2009-11-30 22:08:27:978605 [INFO]   Destroy UniRTSP Agent
2009-11-30 22:08:27:980798 [INFO]   Destroy TCP/MRCPv2 Connection Agent
2009-11-30 22:08:27:982727 [INFO]   Destroy Media Processing Engine
2009-11-30 22:08:27:985232 [INFO]   Destroy MRCP Client

There are also some strange Sofia-SIP "invalid reads". Full valgrind 
traces are attached:
uni-ast-package.valgrind.log.9287 - Trace from first doing "module unload 
res_speech_unimrcp.so" and then "core stop gracefully".
uni-ast-package.valgrind.log.10535 - Trace from "core stop gracefully" 
immediately.

My conclusion is that it is probably Asterisk-related, because if I module 
unload before terminating Asterisk, I don't get a leak. So, this would not 
explain why memory grows with loading and unloading the module - again 
indicating problems in Asterisk, rather than UniMRCP.

Your help/comments would be appreciated.

Kind regards,
Derik Thirion

Reported by thirionjwf on 2009-11-30 20:16:06

achaloyan commented 9 years ago

Reported by achaloyan on 2009-12-01 17:34:18

achaloyan commented 9 years ago 
Hi Derik,

I've played a bit with UniMRCP module load/unload routine ins Asterisk and found 3
issues there

1. Asterisk Connector
I forgot to destroy the logger while unloading module. So file descriptor remained
opened. Fixed in
http://code.google.com/p/unimrcp/source/detail?r=1319

2. UniMRCP tasks
I had a closer look at what valgrind reports running unimrcp server and client even
without Asterisk. It never caused any issue, that's why I didn't pay enough attention
to it before. However there indeed was an issue related to task termination.
Fixed in trunk
http://code.google.com/p/unimrcp/source/detail?r=1320

3. Asterisk crash
This one clearly relates to Asterisk and all the versions are affected. Someone may
want to raise it to Asterisk community too.

Scenario to reproduce
1. Load res_speech_unimrcp module (or any other speech module, it doesn't matter)
2. Do whatever here
3. Unload res_speech_unimrcp
4. Call to Asterisk using the dialplan which use SpeechCreate() without exact name
of
the module, using the default.
5. Asterisk one to one crashes.

The following patch should fix it for Asterisk
--- res_speech.c.org    2007-11-22 03:09:02.000000000 +0400
+++ res_speech.c        2009-12-02 01:51:18.000000000 +0400
@@ -313,7 +313,7 @@
                        /* We have our engine... removed it */
                        AST_RWLIST_REMOVE_CURRENT(list);
                        /* If this was the default engine, we need to pick a new one
*/
-                       if (!default_engine)
+                       if (engine == default_engine)
                                default_engine = AST_RWLIST_FIRST(&engines);
                        ast_verb(2, "Unregistered speech recognition engine '%s'\n",
engine_name);
                        /* All went well */

Please try fixes and provide feedback.
Thanks for the detailed issue report.

Reported by achaloyan on 2009-12-01 17:55:44

achaloyan commented 9 years ago 
Seems it missed also the following to properly terminate tasks/threads
http://code.google.com/p/unimrcp/source/detail?r=1321

Reported by achaloyan on 2009-12-01 19:36:18

achaloyan commented 9 years ago 
Hi there,

I've submitted the patch to the Asterisk community:
https://issues.asterisk.org/view.php?id=16368

Kind regards,
Derik

Reported by thirionjwf on 2009-12-02 08:07:32

achaloyan commented 9 years ago 
This looks fine now. I'll continue testing it.

Reported by thirionjwf on 2010-02-06 21:33:14

achaloyan commented 9 years ago 
Hi there,

Just to let you know - It looks like the bug has now been fixed in Asterisk with the
patch you've supplied.
https://issues.asterisk.org/view.php?id=16368

Kind regards,
Derik

Reported by thirionjwf on 2010-02-19 06:31:26

achaloyan commented 9 years ago 
So, I'm delighted that sanity has prevailed and thanks for the update, Derik.

Reported by achaloyan on 2010-02-19 06:43:26