Investigate supervisor configuration improvements.

[mrmachine]

Seems like it is possible for gunicorn and celery (and other) processes to become orphaned if the system level supervisor is restarted, and the manage.py supervisor process does not successfully shutdown all of its programs and itself within 10 seconds.

Because both supervisors are using the default config which waits for 10 seconds before sending SIGKILL, the system level supervisor might send SIGKILL to manage.py supervisor if any of its programs take longer than 10 seconds to terminate.

One possible solution is to configure system level supervisor with stopwaitsecs = 20 to give manage.py supervisor enough time to gracefully or forcefully stop its programs and terminate itself.

We might also want to use stopasgroup or killasgroup, because both the system level supervisor run programs that start subprocesses (e.g. manage.py supervisor runs gunicorn and celery, which each start their own subprocesses).

[ayeowch]

Adding killasgroup=true for manage.py supervisor is probably a reasonable default and project specific application should specify stopwaitsecs accordingly. For example, we currently use stopwaitsecs=600 for celery to wait for long running tasks.

[ayeowch]

A better solution where we can maintain the timing specified in the project level supervisor config is described here: https://github.com/ixc/deployo/issues/14