Add a SIGTERM handler to ProcessQueue (maybe DBRunner)

jtniehof commented 3 years ago

kill-ing ProcessQueue (i.e. sending SIGTERM) is pretty messy, it just dies directly. It would be nice to have a SIGTERM handler that does a minimal amount of cleanup and leaves the database in a defined state without the currently-processing flag set.

Proposed enhancement

SIGTERM handler that at a minimum:

Kills running processes and cleans up their temp directory, or allows them to complete and does final ingest
Prevents the SIGTERM from propagating to the child processes (ProcessQueue should handle the kill).
Leaves the database in a defined, documented state so it's clear what needs to be done to resume from this point (e.g. what is done about items on the process queue that haven't been processed, and runme objects that have been built but not run; is a reprocessBy required.)
Clears the processing flag, maybe with mention that it was a SIGTERM
Writes to log file that SIGTERM was received
Exits

Alternatives

This is already pretty vague; do-nothing is always an alternative but I always spend a bunch of time hand-cleaning the state if I have to kill a ProcessQueue.

OS, Python version, and dependency version information:

Linux-4.4.0-98-generic-x86_64-with-Ubuntu-16.04-xenial
sys.version_info(major=2, minor=7, micro=12, releaselevel='final', serial=0)
sqlalchemy=1.0.11

Version of dbprocessing

git master (8e5d3ae432fb35227c74bc5615422cf456b39578)

Closure condition

This issue should be closed when: design finalized and PR merged with both the handler and appropriate documentation for how to recover.

balarsen commented 3 years ago

This would be really good. What cleaning do you end up doing? I typically just use the clear processing flag script.

jtniehof commented 3 years ago

Sometimes I'll wind up with the file in the incoming directory and I have to poke around in the database to see if it was ingested and/or added to the processing queue, that sort of thing. Some of this is just going to be homework and documentation: "if the processing is killed this is what you have to do to get going again."

jtniehof commented 3 years ago

Incidentally on a true crash I very much appreciate having the processing flag set since I have this kicked off automatically, so if something crashes out it's good to not have it start back up!

spacepy / dbprocessing