Bad file descriptor during a graceful restart from Cherokee admin

[danielniccoli]

Original author: jrabbit....@googlemail.com (October 26, 2010 23:30:12)

What steps will reproduce the problem?

1. Run cherokee
2. Run cherokee-admin
3. Change any setting
4. Click save
5. Perform a graceful restart

What is the expected output? What do you see instead?

The following appears in the ssh console:

[root@server1 html]# PID 9699: exited re=0 Handling Graceful Restart.. 27/10/2010 00:25:48.089 fdpoll-epoll.c:140 - epoll_ctl: ep_fd 13, fd 0: 'Bad file descriptor' file thread.c:1872 (watch_accept_MULTI_THREAD): this should not happen PID 9699: exited re=0 Cherokee Web Server 1.0.9 (Oct 26 2010): Listening on port ALL:81, chrooted, TLS disabled, IPv6 disabled, using epoll, 4096 fds system limit, max. 2041 connections, caching I/O, 20 threads, 102 connections per thread, standard scheduling policy

If I do a hard restart, I do not get any error message:

[root@server1 html]# Server is exiting.. PID 9764: exited re=0 Cherokee Web Server 1.0.9 (Oct 26 2010): Listening on port ALL:81, chrooted, TLS disabled, IPv6 disabled, using epoll, 4096 fds system limit, max. 2041 connections, caching I/O, 20 threads, 102 connections per thread, standard scheduling policy

This is on a test virtual machine - there is no traffic hitting the server at the time of the restart.

What version of the product are you using? On what operating system?

1.0.9b5610 on Centos 5.5 x64

Original issue: http://code.google.com/p/cherokee/issues/detail?id=1038

[danielniccoli]

From jrabbit....@googlemail.com on October 26, 2010 23:38:28 This isn't specific to cherokee admin - sending a SIGHUP to cherokee gives the same message.

[danielniccoli]

From usebox...@gmail.com on November 14, 2010 12:41:23 Same here, Fedora 14 64bits:

Handling Graceful Restart.. 14/11/2010 12:40:17.868 fdpoll-epoll.c:140 - epoll_ctl: ep_fd 7, fd 3: 'Bad file descriptor' file thread.c:1872 (watch_accept_MULTI_THREAD): this should not happen PID 3188: exited re=0

[danielniccoli]

From usebox...@gmail.com on November 14, 2010 13:00:37 I'm attaching a trace log.

Seems that the same file descriptor is being freed several times (first and successful one is at cherokee_server_handle_HUP).

Seems that after that, it's being freed again at cherokee_server_free.

[danielniccoli]

From usebox...@gmail.com on November 14, 2010 13:22:41 I don't know if this patch is simply wrong, but the idea is to do some cleaning in the HUP function instead of just closing the fds.

[danielniccoli]

From alobbs on November 14, 2010 16:00:39 Thanks for the patch Juanjo. I'm afraid there is a problem about it though.

There might be other threads accessing those cherokee_bind_t objects at the time the signal is catch by the thread executing these free()s. That would most likely endup in the whole server crashing.

[danielniccoli]

From usebox...@gmail.com on November 14, 2010 16:07:32 It's possible, forget it :)

Anyway, there's nothing wrong in this "bug" (if the fd it already closed, it's OK as fas as I've been checking the source).

I guess it's just the message, that it's misleading.

[danielniccoli]

From jrab...@tthfanfic.org on November 14, 2010 21:38:51 Would not setting the fd to -1 after closing it then checking if it is -1 before trying to close it again not resolve things safely?

[danielniccoli]

From leon.dig...@gmail.com on February 16, 2011 20:11:52 I can reproduce this with Debian Squeeze and Cherokee 1.0.20

{ 'type': "error", 'time': "16/02/2011 18:57:31.346", 'title': "epoll_ctl: ep_fd 15, fd 3: 'Bad file descriptor'", 'code': "fdpoll-epoll.c:140", 'error': "81", 'description': "The issue seems to be related to your system.", 'version': "1.0.20", 'compilation_date': "Feb 8 2011 09:50:44", 'configure_args': " '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS=' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS='", 'backtrace': "" }

{ 'type': "error", 'time': "16/02/2011 19:57:09.331", 'title': "epoll_ctl: ep_fd 19, fd 3: 'Bad file descriptor'", 'code': "fdpoll-epoll.c:140", 'error': "81", 'description': "The issue seems to be related to your system.", 'version': "1.0.20", 'compilation_date': "Feb 8 2011 09:50:44", 'configure_args': " '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS=' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS='", 'backtrace': "" }

{ 'type': "error", 'time': "16/02/2011 20:01:32.097", 'title': "epoll_ctl: ep_fd 9, fd 3: 'Bad file descriptor'", 'code': "fdpoll-epoll.c:140", 'error': "81", 'description': "The issue seems to be related to your system.", 'version': "1.0.20", 'compilation_date': "Feb 8 2011 09:50:44", 'configure_args': " '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS=' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS='", 'backtrace': "" }

{ 'type': "error", 'time': "16/02/2011 20:03:17.971", 'title': "epoll_ctl: ep_fd 25, fd 3: 'Bad file descriptor'", 'code': "fdpoll-epoll.c:140", 'error': "81", 'description': "The issue seems to be related to your system.", 'version': "1.0.20", 'compilation_date': "Feb 8 2011 09:50:44", 'configure_args': " '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS=' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--libexecdir=${prefix}/lib/cherokee' '--disable-maintainer-mode' '--disable-dependency-tracking' '--host=x86_64-linux-gnu' '--build=x86_64-linux-gnu' '--enable-os-string=Debian GNU/Linux' '--prefix=/usr' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--docdir=/usr/share/doc/cherokee-doc' '--with-wwwroot=/var/www' '--enable-pthread' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=' 'CPPFLAGS='", 'backtrace': "" }