Closed Cyberboss closed 1 year ago
Also the .Completion
event won't fire if there are items still in the channel https://stackoverflow.com/a/66521303
Quite the peculiar set of issues, that's for sure. I've been hoping to rewrite mass swaths of Remora's codebase[1][2], so I suppose I'll definitely tack this onto the list; rewriting dispatch would certainly be a worthwhile investment based on my investigations[3]
1 #254 2 Remora.Commands PR 3 Discussion in the Discord
This hacky reflection workaround was able to solve our issue: https://github.com/tgstation/tgstation-server/pull/1509
Interesting - thank you for the in-depth analysis. I've pushed a suggested fix to the deadlock-fix
branch, which should alleviate the problem and fix another issue I uncovered. Please take a look when you have time.
I've integrated this in my software and haven't heard the same bug report, so it looks to be working
Excellent!
Description
The
_finalizer
Task
relies on the_dispatcher
Task
closing the channel to be able to complete. But it's possible for this to never happen.The channel completion call relies on several throws possible from the CancellationToken not happening.
The deadlock is usually avoided because the
_finalizer
Task
also usually throws out before it gets to that pointAll the lines listed can throw
TaskCanceledException
s. The deadlock occurs if any of the first group throw and none of the second group throw. At that point, the app will halt forever when it awaits the Reader's completion.Steps to Reproduce
It's very difficult to reproduce. The only example I have is a dump of https://github.com/tgstation/tgstation-server deadlocked at this point. For security reasons (bot tokens and SQL credentials) I cannot share it publicly.
Expected Behavior
ResponderDispatchService.DisposeAsync() should complete eventually.
Current Behavior
ResponderDispatchService.DisposeAsync() can deadlock waiting for a channel completion event that never comes.
Library / Runtime Information
dotnet: 6.0 Remora.Discord: 2022.48.0 tgstation-server: 5.12.4 OS: Windows Server 2022