Open Sentynel opened 10 months ago
Thank you for the excellent investigation and writeup :)
This is partially fixed in tobi's above PR ^
Though as has been determined, there's still a possible issue that enqueuing async work from within a worker function can cause a deadlock if the worker queues are currently full. This is a fix I'm working on for the go-runners
package.
Removing this from the current milestone since we fixed the most egregious example for 0.13.0; we can work on a more thorough fix for 0.14.0.
Describe the bug with a clear and concise description of what the bug is.
Importing a domain blocklist covering several hundred domains causes my GtS server to stop processing any tasks on the worker pool. Processing of the domain blocklist ceases part way through, and subsequent incoming statuses etc are not processed.
This is because the way we handle domain blocks uses the worker pool queue APIs in an unsafe way and can deadlock if the queue fills up. A stack trace of one of the workers while stuck looks like this:
Note that
deleteAccountStatuses()
, which loops over all statuses associated with the account we're currently deleting and queues new tasks for removing them all, is callingMustEnqueueCtx()
(viaEnqueueClientAPI()
). This is unsafe because we're in a worker context at the point we call this, and this inMustEnqueueCtx()
we attempt to push a new function into a buffered channel of finite size with our own function in it. Do this to all of the workers simultaneously, and we deadlock with all workers waiting for somebody to pop a function off the worker pool to free space.We should ensure that we don't call
MustEnqueueCtx()
from a worker context. In this particular case we can solve the issue by processing status deletions from our own task rather than enqueuing new ones to avoid flooding the task queue. This also happens for follows processing in the same file.What's your GoToSocial Version?
v0.12.0
GoToSocial Arch
No response
What happened?
No response
What you expected to happen?
No response
How to reproduce it?
No response
Anything else we need to know?
No response