[BUG] Webdis exits immediately on start on linux x86_64

[agentzh]

Hello!

I'm trying to use webdis on linux x86_64 with the latest git master HEAD (28ec18582a2354162a460713b09553e3a5276b7a) and tries to start the ./webdis executable with the default webdis.json file in the repos.

But ./webdis immediately exits at startup, and nothing is listening on the port 7379. turning off the daemonize option in webdis.json gives the same result.

the last few lines of the strace output look like this:

mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f342ce70000
mprotect(0x7f342ce70000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f342d66fff0,     flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f342d6709e0, tls=0x7f342d670710, child_tidptr=0x7f342d6709e0) = 3999
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 23
setsockopt(23, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
fcntl(23, F_SETFD, 0x800 /* FD_??? */)  = 0
bind(23, {sa_family=AF_INET, sin_port=htons(7379), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(23, 128)                         = 0
exit_group(0)                           = ?

The output of uname -a:

Linux agentzh-thinkpad 2.6.33.1 #2 SMP Fri Mar 26 02:09:08 CDT 2010 x86_64 Intel(R) Core(TM)2 Duo CPU T9600 @ 2.80GHz GenuineIntel GNU/Linux

The output of cc -v

$ cc -v
Reading specs from /usr/lib64/gcc/x86_64-slackware-linux/4.4.3/specs
Target: x86_64-slackware-linux
Configured with: ../gcc-4.4.3/configure --prefix=/usr --libdir=/usr/lib64 --enable-shared --enable-bootstrap --enable-languages=ada,c,c++,fortran,java,objc --enable-threads=posix --enable-checking=release --with-system-zlib --with-python-dir=/lib64/python2.6/site-packages --disable-libunwind-exceptions --enable-__cxa_atexit --enable-libssp --with-gnu-ld --verbose --disable-multilib --target=x86_64-slackware-linux --build=x86_64-slackware-linux --host=x86_64-slackware-linux
Thread model: posix
gcc version 4.4.3 (GCC) 

Any clues?

[agentzh]

The webdis.log file is empty, BTW.

[nicolasff]

Hello!

Thanks for this detailed report. I'm not sure what could be wrong, bind and listen seem to have succeeded. Which version of libevent are you using? Could you paste the output of ldd ./webdis?

Thanks.

[agentzh]

Hello!

Which version of libevent are you using?

I'm just using the libevent shipped with my linux distribution. It seems to be 1.4.x. Should I upgade?

Could you paste the output of ldd ./webdis?

$ ldd ./webdis linux-vdso.so.1 => (0x00007fff559a6000) libevent-1.4.so.2 => /usr/lib64/libevent-1.4.so.2 (0x00007f213c397000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f213c17a000) libc.so.6 => /lib64/libc.so.6 (0x00007f213be09000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f213bbef000) librt.so.1 => /lib64/librt.so.1 (0x00007f213b9e7000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f213b7ce000) /lib64/ld-linux-x86-64.so.2 (0x00007f213c5b1000)

Regards, -agentzh

[nicolasff]

Hi again,

I've straced it as well and it seems to fail in calling event_add for you which points to an issue with libevent. I have added a log message if event_add returns -1 but unfortunately libevent will call exit(1) for serious failures so I'm not sure that it will be caught. Could you please try the latest master?

If you look in server.c at the bottom, you will see the call to event_add, which should show up in your strace as epoll_ctl(... EPOLL_CTL_ADD ...); if you see an exit before then it looks like libevent could not add the file descriptor to its backend.

Could you please try it with libevent 2, or on a more recent system?

Thanks a lot for reporting this, and for your interest. I know your redis plugin and would be very interested in discussing comparisons between the two.

Regards,

Nicolas

[agentzh]

Hello!

On Mon, Jun 18, 2012 at 2:33 AM, Nicolas Favre-Felix reply@reply.github.com wrote:

I've straced it as well and it seems to fail in calling event_add for you which points to an issue with libevent. I have added a log message if event_add returns -1 but unfortunately libevent will call exit(1) for serious failures so I'm not sure that it will be caught. Could you please try the latest master?

With the latest master HEAD, I got the same result. It seems that webdis is using somet APIs that are missing in libevent 1.4, like event_base_new. I've upgraded libevent to 2.0.19 and it is finally working now.

Maybe you should make this clear in README or Makefile?

Thanks a lot for reporting this, and for your interest. I know your redis plugin and would be very interested in discussing comparisons between the two.

I've just benchmarked the simple GET command with:

ab -k -c50 -n200000 http://127.0.0.1:7379/GET/hello

It got 22% faster than nginx + ngx_lua + lua-resty-redis configured like below:

location /t {
    content_by_lua '
        local cjson = require "cjson"
        local redis = require "resty.redis"
        local red = redis:new()

        red:set_timeout(1000) -- 1 sec

        local ok, err = red:connect("127.0.0.1", $TEST_NGINX_REDIS_PORT)
        if not ok then
            ngx.log(ngx.ERR, "failed to connect: ", err)
            return ngx.exit(503)
        end

        local res, err = red:get("hello")
        if err then
            ngx.say("failed to get dog: ", err)
            return
        end

        if not res then
            return ngx.exit(404)
        end

        ngx.say(cjson.encode({ hello = res }))

        red:set_keepalive(0, 1000)
    ';
}

Because lua-resty-redis is a redis driver implemented in pure Lua (based on ngx_lua's nonblocking cosocket API) and is more generic in terms of functionalities than webdis, I'm not very surprised that webdis is a little faster in this tiny example :)

BTW, I've noticed another issue in webdis though. When webdis is running, if I kill the redis process and then start it again, webdis will keep returning the 503 error page, which is certainly unacceptable. Sorry that I'm too lazy to create a separate github issue for this :P

Best regards, -agentzh

[nicolasff]

Thanks for this test. I thought it might be event_base_new() so I specifically looked for it in the libevent documentation and it was present there. I suspect it might be that the online docs are only for the “latest 1.4.x” and that you have a 1.4 without a working event_base_new().

I'm not surprised that C is a little faster than Lua, but I guess we're in the same ballpark. I've created a separate ticket to track the issue with 503s, will work on this soon.

Thanks!

Nicolas