Improve thread pool error handling

[jeffschoner]

Summary

All errors and exceptions coming out of a thread pool job are now logged and sent to the error handler

If any error does reach the top of the stack on a thread pool thread, it will now crash the process rather than silently kill the thread. Because StandardError is rescued in the TaskProcessor, this impacts only severe errors like NoMemoryError or SecurityError raised by activity or workflow code, or ordinary error raised due to bugs in temporal-ruby itself.

It's perhaps somewhat controversial to crash the worker process, but Exception raised out of user code or unexpected errors in the pollers, task processors, or thread pools, leave the worker in an unknown state, where it's unclear that it can continue to safely process work.

Motivation

Before this change, when an activity or workflow task raises an error that is not a subclass of StandardError, it silently kills the thread pool thread it is running on. Additionally, any errors in the ensure/rescue blocks in the pollers, task processors or the thread pool will cause this same behavior. Eventually, workers can run out of thread pool threads and become "zombie" workers that will stop polling for tasks, all while logging no errors.

For example, I've seen this occur when an error raised by an activity contains a circular reference. When Oj tries to serialize it, it will run out of memory and raise NoMemoryError. This is a subclass of Exception not StandardError, and therefore is not caught by an ordinary rescue => e clause. Eventually this reaches the top of the activity task thread pool thread and causes it to silently exit before it can increment the available_threads and signal the availability condition variable to free up resources. Memory frees up at this point, so the worker continues to run, but other failures may have occurred on other threads in the process during this period of memory exhaustion, leaving the worker in an unknown state. The activity will then be retried after it times out, which in turn kills another thread on one of the workers. If this happens enough times between worker service restarts, the entire worker fleet will not be able to process any activities.

Testing

There are new specs for these thread pool cases in both regular and scheduled thread pools. Some other specs also had to be updated because they had errors throwing on a background thread that started surfacing hidden failures.

[jeffschoner]

This change has also been running within Stripe since around July with good results

[DeRauk]

Thanks for this PR @jeffschoner. The example silent failure you gave in the description sounds nasty and tough to debug.