Copterfly / modwsgi

Automatically exported from code.google.com/p/modwsgi
0 stars 0 forks source link

Daemon process not shutdown cleanly on 'graceful' restart/stop. #102

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
When a 'graceful' restart or a 'graceful-stop' is done, any daemon processes do 
not appear to be 
shutdown cleanly but are instead forcibly killed off. This was first raised as 
possibility in issue #95.

Original issue reported on code.google.com by Graham.Dumpleton@gmail.com on 29 Aug 2008 at 7:45

GoogleCodeExporter commented 9 years ago
Hmmm, maybe I am partly wrong about this, for 'graceful' restart do get:

[Thu Sep 18 17:52:24 2008] [notice] SIGUSR1 received.  Doing graceful restart
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2619): Shutdown requested 
'wsgi'.
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2619): Stopping process 'wsgi'.
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2619): Cleanup interpreter ''.
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2620): Cleanup interpreter ''.
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2621): Cleanup interpreter ''.
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2619): Terminating Python.
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2621): Terminating Python.
[Thu Sep 18 17:52:24 2008] [info] mod_wsgi (pid=2620): Terminating Python.

So, does seem okay there after all. At least for self compiled Apache on Mac OS 
X (Tiger).

But then for 'graceful-stop' get:

[Thu Sep 18 17:52:49 2008] [info] removed PID file 
/usr/local/apache-2.2.4/logs/httpd.pid (pid=2617)
[Thu Sep 18 17:52:49 2008] [notice] caught SIGWINCH, shutting down gracefully
[Thu Sep 18 17:52:49 2008] [info] mod_wsgi (pid=2627): Cleanup interpreter ''.
[Thu Sep 18 17:52:49 2008] [info] mod_wsgi (pid=2628): Cleanup interpreter ''.

So, in that case processes must just be getting killed off.

Original comment by Graham.Dumpleton@gmail.com on 18 Sep 2008 at 7:57

GoogleCodeExporter commented 9 years ago
But in run where we just 'start' and then 'graceful-stop' it does shutdown 
properly.

[Thu Sep 18 18:01:41 2008] [info] removed PID file 
/usr/local/apache-2.2.4/logs/httpd.pid (pid=3573)
[Thu Sep 18 18:01:41 2008] [notice] caught SIGWINCH, shutting down gracefully
[Thu Sep 18 18:01:41 2008] [info] mod_wsgi (pid=3576): Cleanup interpreter ''.
[Thu Sep 18 18:01:41 2008] [info] mod_wsgi (pid=3577): Cleanup interpreter ''.
[Thu Sep 18 18:01:41 2008] [info] mod_wsgi (pid=3576): Terminating Python.
[Thu Sep 18 18:01:41 2008] [info] mod_wsgi (pid=3577): Terminating Python.
[Thu Sep 18 18:01:42 2008] [info] mod_wsgi (pid=3575): Shutdown requested 
'wsgi'.
[Thu Sep 18 18:01:42 2008] [info] mod_wsgi (pid=3575): Stopping process 'wsgi'.
[Thu Sep 18 18:01:42 2008] [info] mod_wsgi (pid=3575): Cleanup interpreter ''.
[Thu Sep 18 18:01:42 2008] [info] mod_wsgi (pid=3575): Terminating Python.

So, problem case appears to be for 'start', 'graceful' and then 
'graceful-stop'. 

Also appears to be a problem with 'start', 'graceful' and then 'stop'. In this 
latter case, also don't see daemon process 
shutdown properly.

Issue seems to be that 'graceful' causes problems later on.

Original comment by Graham.Dumpleton@gmail.com on 18 Sep 2008 at 8:04

GoogleCodeExporter commented 9 years ago
This all seems to work fine when look at it again.

Couple of things could have been going on here. The first is that working now 
with changes to how signals handled 
in daemon processes. The second is that logging just wasn't ending up where it 
was meant to previously.

One thing of note though. If you do a graceful restart, the worker process you 
may just have made a request via 
will be hanging around until keep alive timeout expires. The daemon processes 
though get shutdown immediately. 
This means that the worker process has a name of a WSGI socket file which is no 
longer valid. Thus can get a Service Temporarily Unavailable error response. 
Errors in Apache log are:

[Thu Mar 12 21:10:01 2009] [error] [client ::1] (2)No such file or directory: 
mod_wsgi (pid=53225): Unable to 
connect to WSGI daemon process 'hello' on 
'/usr/local/apache-2.2.11/logs/wsgi.53186.4.1.sock' after multiple 
attempts.

Not sure there is much that can be done about that. When have transient daemon 
support, may be able to go back 
and rerequest via monitor process for name of WSGI socket, but still will not 
help if the daemon was a static one 
and was shutdown due to configuration changing. Anyway, can deal with that 
later.

Original comment by Graham.Dumpleton@gmail.com on 12 Mar 2009 at 10:12

GoogleCodeExporter commented 9 years ago
Hello,

Was the "Unable to connect to WSGI daemon process" error mentioned above ever 
improved on? My server is generating errors like this every hour when I do a 
graceful restart of Apache (which I do so I can process logs for reports).

Longer term the solution is a different architecture but if there was a fix for 
this issue it would be great.

Thanks,
AJ

Original comment by aj.oster...@googlemail.com on 15 Jul 2011 at 5:46

GoogleCodeExporter commented 9 years ago
Reading comment 3 one more time (earlier in the day so a little more awake) it 
almost sounds like I should stop use 'graceful' and just do a proper hard stop 
and start - at least that way no processes will be hanging about waiting for a 
timeout ...

Original comment by aj.oster...@googlemail.com on 16 Jul 2011 at 9:02

GoogleCodeExporter commented 9 years ago
Have you looked at using a piped log handler with Apache and having the 
external log handler handle log file rotation automatically rather than 
restarting Apache so can cause log files to be moved out of the way?

Original comment by Graham.Dumpleton@gmail.com on 19 Jul 2011 at 4:20