Suggestions on getting close called on storage on process termination?

[jimfulton]

RelStorage writes its persistent cache files on close, but most apps don't bother to close their databases on close, because there was no reason to bother before.

How do you handle this?

I wonder if ZODB should grow an option to register an atexit handler to close the database on process exit. Even then a program would need to set up a signal handler to handle kills.

[jamadden]

We've found it necessary to tear down certain resources at process exit for a long time, so this was no stretch for us. That includes things like Redis connections and associated locks; open Whoosh indexes that may need to flush changes; notifying other members of the cluster that we're going down, etc. (I suspect many applications may have similar needs.)

I agree it could be nice to have a ZODB-specific pattern for this, though---closing isn't just good for RelStorage; it's good to close a FileStorage when you're done with it, for example, so that the locks get deleted and the index gets updated.

In one of our WSGI apps, we're using gunicorn. We manually run the server code, so when the main loop exits (for whatever reason---gunicorn uses signals to manage worker processes, so it's usually a KeyboardInterrupt), we tear things down. Something like this:

try:
   http_server.serve_forever()
finally:
   server.close()

It's not perfect, but it seems to be mostly sufficient.

[jimfulton]

On Mon, May 1, 2017 at 7:29 AM, Jason Madden notifications@github.com wrote:

We've found it necessary to tear down certain resources at process exit for a long time, so this was no stretch for us. That includes things like Redis connections and associated locks; open Whoosh indexes that may need to flush changes; notifying other members of the cluster that we're going down, etc. (I suspect many applications may have similar needs.)

But how these needs are met is fairly application specific. For example, at ZC, we used ZooKeeper to notify clients that servers has gone away or moved.

I agree it could be nice to have a ZODB-specific pattern for this, though---closing isn't just good for RelStorage; it's good to close a FileStorage when you're done with it, for example, so that the locks get deleted

Locks should not be deleted. (I realized in writing this that zc.lockfile has a bug. It should create the lock file with the exclusive flag. It doesn't now.)

and the index gets updated.

ZEO takes care of this now.

In one of our WSGI apps, we're using gunicorn. We manually run the server code, so when the main loop exits (for whatever reason---gunicorn uses signals to manage worker processes, so it's usually a KeyboardInterrupt), we tear things down. Something like this:

try: http_server.serve_forever() finally: server.close()

It's not perfect, but it seems to be mostly sufficient.

I ended up adding a signal handler, because when supervisord (probably zdaemon too, but I didn't check) stops a process, it sends a sigterm and atexit handlers aren't called:

https://github.com/karlproject/karl/blob/master/karl/application.py#L364

IMO, this should be easier.