Dispatcher failures may leave coroutines uncompleted

[dkhalanskyjb]

Describe the bug

During the review of https://github.com/Kotlin/kotlinx.coroutines/pull/4181, it became evident that we don't properly handle failures in coroutine dispatchers, and this can surface in ways other than just strange-looking exceptions.

Provide a Reproducer

val dispatcher = newSingleThreadContext("unreliable friend")
runTest {
    launch(dispatcher, start = CoroutineStart.UNDISPATCHED) {
        try {
            println("This code runs...")
            suspendCancellableCoroutine<Int> { cont ->
                // we launch a separate thread and wait a bit,
                // because we want the coroutine to actually suspend
                // and go through a dispatch.
                launch(Dispatchers.Default) {
                    delay(100)
                    // close the dispatcher, now it will throw on `dispatch`
                    dispatcher.close()
                    // try dispatching the coroutine
                    cont.resume(3)
                }
            }
        } catch (e: Throwable) {
            println("Caught $e")
            throw e
        } finally {
            println("... therefore, this code must run.")
        }
    }
}

This code will hang after printing This code runs..., as the launched coroutine never finishes.

[dkhalanskyjb]

It's unclear what to do in this scenario, though. Here are some options:

• Try to fail the whole program by calling handleCoroutineException. We don't call it a mechanism of last-resort exception propagation for nothing.
• On the JVM, Executor.asCoroutineDispatcher tries cancelling the task and redispatching it to Dispatchers.IO if the executor rejected it. We can't do literally that in common code, as there is no Dispatchers.IO on platforms with no threads, but if there are no threads, we can just use DefaultExecutor. The problem is, what if finalizers contain some code that must run on a specific thread? If Dispatchers.Main fails, it's usually a contract violation to try to run its code on Dispatchers.IO.
• We can pretend that the coroutine that couldn't get dispatched did finish with an error. This way, finalizers won't be run (which is a contract violation), but at least the whole coroutine hierarchy won't deadlock because of an incorrect dispatch.
• Something else?