crash at end of rtsp session when using pocketsphinx plugin

[GoogleCodeExporter]

When using pocketsphinx plugin with unirtsp, unimrcpserver crashes at the end 
of the call/session about 30-40% of the time.

When using the gdb debugger, backtrace shows that it always occurs during a 
call to apr_pool_destroy(), and either occurs in one of the following two 
places:
* rtsp_server.c:319, where rtsp_server_session_destroy invokes 
apr_pool_destroy()
* mrcp_pocketsphinx.c:833, where pocketsphinx_recognizer_run invokes 
apr_thread_exit() which ultimately invokes apr_pool_destroy()

To reproduce:
1. Configure UniMRCP to use pocketsphinx plugin and unirtsp (mrcpv1).
2. Run any simple umc scenario, or use Voxeo Prophesy browser to place a call.  
(Any simple recognition request with a simple English grammar and language 
model.)
3. At end of call/session, unimrcpserver will sometimes crash (happens about 
30% of the time for us).

Version info:
unimrcp-0.10.0
apr-1.4.2
apr-util-1.3.9
sphinxbase-0.6
pocketsphinx-0.6
unimrcp-deps-0.9.0
OS: Linux centoskde1 2.6.18-194.11.1.el5xen

I will attach sample output soon, which includes gdb debugger info, for all the 
following configurations/scenarios:
* crash with umc in rtsp_server.c
* crash with umc in mrcp_pocketsphinx.c
* crash with voxeo in rtsp_server.c
* crash with voxeo in mrcp_pocketsphinx.c

Additional info:
* it never crashes for us when using demo asr plugin, so it seems to be 
specific to the pocketsphinx plugin
* it crashes during different timing of hangups, e.g. during prompt, after 
prompt, after recognition, beginning of call, etc.  short prompts, long 
prompts, etc.
* it crashes with multiple clients (umc and voxeo), so it doesn't seem to be 
specific to a client/browser
* we tried modifying makefiles so that the linker uses -pthread and not 
-lpthread option, but libtool automatically adds the -lpthread option, and we 
don't know of a workaround without manually linking

Have you encountered this issue?  Are there any production deployments that use 
pocketsphinx plugin (or any other plugin using separate recognition threads) 
that are successful?

Thanks in advance!

"Dexter"

Original issue reported on code.google.com by dexterou...@gmail.com on 24 Aug 2010 at 3:39

[GoogleCodeExporter]

Sorry, I meant to give you the following OS info:
CentOS release 5.5 (Final)
(according to /etc/redhat-release)

Original comment by dexterou...@gmail.com on 24 Aug 2010 at 3:43

[GoogleCodeExporter]

Original comment by dexterou...@gmail.com on 24 Aug 2010 at 3:50

Attachments:

• Aug23-12.54.14-rtspcrash-voxeo-psphinx-example1.log
• Aug23-12.59.00-rtspcrash-voxeo-psphinx-example2.log
• Aug23-13.09.29-psplugincrash-voxeo-psphinx-example1.log
• Aug23-13.12.17-psplugincrash-voxeo-psphinx-example2.log
• Aug23-14.00.47-rtspcrash-umc-psphinx-example1.log
• Aug23-14.05.43-psplugincrash-umc-psphinx-example1.log

[GoogleCodeExporter]

Hi Dexter,

Thanks for the detailed description of the problem.

I tried to reproduce and eventually got the same, although I was not so "lucky" 
and it crashed only after 20 or so attempts. Now looking at the source code of 
the plugin and the backtraces you provided, it seems I know where the problem 
is in.

As you properly noticed, apr_thread_exit() destroys memory pool of its thread. 
And this pool is a child of the pool associated with the RTSP/MRCP session. So, 
it causes a typical bug that a child memory pool (apr_thread_exit) is destroyed 
after destruction of its parent memory pool (rtsp_server_session_destroy). I 
don't know why it happens so frequently to you, but I admit the issue exists.

It's clear that mrcp_engine_channel_close_respond() MUST be called after 
apr_thread_exit()... 

Original comment by achalo...@gmail.com on 25 Aug 2010 at 11:59

• Changed state: Accepted
• Added labels: Component-Plugin, OpSys-All, Priority-Medium, Type-Defect

[GoogleCodeExporter]

I've just fixed the issue in r1773.

Unfortunately, I don't have the time to test this fix enough, and honestly 
speaking, I use PocketSphinx very rarely just for basic tests. So, give it a 
try and come back with the results, please.

Original comment by achalo...@gmail.com on 25 Aug 2010 at 12:26

• Changed state: Fixed

[GoogleCodeExporter]

Excellent!  It definitely seems to have fixed the problem, so you are welcome 
to close this issue.

(I had actually considered adding the thread join, but I think I unconsciously 
put it low on my mental list of "things to try" because I was concerned that 
the recognizer thread might take a while to terminate.  But it turns out that 
it's really quick, so I think we're going to go with this solution.)

Thanks so much for the quick turnaround, especially since it's in a plugin/demo 
and not part of the actual mrcp-server.

Original comment by dexterou...@gmail.com on 27 Aug 2010 at 8:38

[GoogleCodeExporter]

You're welcome and thanks for the issue report!

Original comment by achalo...@gmail.com on 1 Sep 2010 at 9:56

• Changed state: Verified