gearman / gearmand

http://gearman.org/
Other
741 stars 138 forks source link

missing initialization of libevent for use with multiple threads #262

Closed zvpunry closed 4 years ago

zvpunry commented 4 years ago

Libevent functions must not be called simultaneously by multiple threads, unless libevent's thread support has been initialized by evthread_use_pthreads() or evthread_use_windows_threads(). This is mentioned under "Library setup" in http://libevent.org/doc/

When these functions are not called before libevent is used, it will just access its internal data structures without trying to prevent any data races.

I suspect this bug to be the cause for sporadic failures of gearman to react to data on open filehandles. Some clients send request or ascii commands like status and don't get an answer. In fact, gearmand completely ignores input on some of its filehandles. It took about 4 weeks before this bug appeared and after a restart is was gone for some weeks. After I added "--threads=32" (default: 4) to /etc/default/gearman-job-server the bug appeared after three days and then I put a "--threads=1" into the config and it seemed to disappear.

This is when I looked at the source and found that bug.

Another test was using LD_PRELOAD= to call evthread_use_pthreads() before anything else. And then I started it with --threads=32 and the problem didn't happen anymore.

/*
 * gcc -levent -levent_pthreads -lpthread -fPIC -shared -o init_evthread_use_pthread.so init_evthread_use_pthread.c
 *
 * Modify the /etc/init.d/gearman-job-server script to use
 * LD_PRELOAD=/root/scripts/init_evthread_use_pthread.so
 *
 * look at the start() function, the start should look like:
 *
 * start-stop-daemon --start \
 *  --exec /usr/bin/env "LD_PRELOAD=/root/scripts/init_evthread_use_pthread.so" $DAEMON -- \
 *  --pid-file=$PIDFILE \
 *  --user=$GEARMANUSER \
 *  $PARAMS
 *
 */

#include <event2/event.h>
#include <event2/thread.h>
#include <stdlib.h>

static void __attribute((constructor)) init()
{
    evthread_use_pthreads();
    unsetenv("LD_PRELOAD");
}

Proposed fix: Add an early unconditional call to evthread_use_pthreads() to gearmand/gearmand.cc in function main().

riBoon commented 4 years ago

Faced this problem too. This here helps fixing it on my own. Thanks for you efforts in this!