Fiber's interrupt completes only once the fiber is stopped, and because in ListenerDriver it's called by unsafe.run it means that it will block the current thread until it's over. Interrupt will trigger the finalizer from onExit, which calls call.close which needs to run on the gRPC threadpool (the same where the current thread is being blocked).
If you receive a lot of cancel calls at the same time, all threads might end up being blocked. It's ok when your threadpool is unbounded (which is the default), but when using a bounded threadpool (fixed or ForkJoin) as recommended in the docs, this is a deadlock.
See the attached screenshot: using ForkJoinPool, when we receive a lot of stream disconnections, all threads end up being blocked on the unsafe.run of fiber.interrupt.
This PR changes it to fiber.interruptFork to prevent being blocked while the fiber is interrupted.
thesamet
commented
3 months ago
Fiber's
interruptcompletes only once the fiber is stopped, and because inListenerDriverit's called byunsafe.runit means that it will block the current thread until it's over.Interruptwill trigger the finalizer fromonExit, which callscall.closewhich needs to run on the gRPC threadpool (the same where the current thread is being blocked).If you receive a lot of cancel calls at the same time, all threads might end up being blocked. It's ok when your threadpool is unbounded (which is the default), but when using a bounded threadpool (fixed or ForkJoin) as recommended in the docs, this is a deadlock.
See the attached screenshot: using
ForkJoinPool, when we receive a lot of stream disconnections, all threads end up being blocked on theunsafe.runoffiber.interrupt.This PR changes it to
fiber.interruptForkto prevent being blocked while the fiber is interrupted.