Improve exception handling for connection errors or add signal handling

[lost-theory]

Occasionally when killing a scripterator job on the command line it will start throwing thousands of these errors until it's forcefully killed again:

Error: Record 5200380: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200381: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200382: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200383: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200384: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200385: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200386: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200387: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...
Error: Record 5200388: Mysql2::Error: closed MySQL connection: DELETE FROM `sometable` WHERE ...

I'm guessing the different behavior depends on whether the kill signal is received inside or outside of this block:

https://github.com/lumoslabs/scripterator/blob/23f6560/lib/scripterator/runner.rb#L122-L132

When kill is received the DB connection is closed and an exception is raised, but it continues running forever because it's catching all exceptions.

When this happens we have to do a bunch of manual cleanup (deleting IDs from the checked and failed sets and retrying the failed IDs, filtering out these connection errors vs. legit errors).

We should either change the bare rescue clause to die on DB connection errors or add signal handling to stop iterating over records when SIGTERM is received.

[cfurrow]

Thanks for reporting this. I'll give it a look soon. I think I'll definitely take the route of handling db connection exceptions differently.