job validator hang at exit

[grondo]

On a production system we've seen cases of the job validator hanging at exit. It appears that at the time there are multiple threads active, and the exiting main thread is blocked on a lock, possibly waiting for these threads.

Perhaps there is something unsafe that the validator (and likewise frobnicator) is doing with concurrent.futures.

[grondo]

Ah, probably the validator should be calling concurrent.futures.shutdown(wait=False, cancel_futures=True) before exiting.