Closed danielmitterdorfer closed 4 years ago
Thank you for the report and the debug log to help with looking at this. Having looked through the log, it looks like the remote Actor closed the socket at some point while there were still message queue to send to that remote queued, and the send got attempted. It's entirely possible for sockets to close, and the TCPTransport should accomodate those situations, but as you noted, it wasn't properly handling the ValueError
scenario where the socket fd was invalid.
I've uploaded a fix (https://github.com/kquick/Thespian/commit/a99fbf92590c470f3b7cec74e448fa5d5aa928c2) that should properly handle the ValueError
scenario. It's a little tough to reproduce this with local tests, but it sounds like it's not too difficult for you to reproduce, so I'm hoping you can confirm that this helps.
It's entirely possible that there's some other underlying issue that resulted in the socket disconnect; please check that the updated version is giving you the expected results and let me know if there's something else I can help with here.
[Note: apparently pushing a commit with the word "Fix" followed by a hash and a bug number causes github to automatically close an issue. TIL. I've re-opened the issue to wait for your confirmation and further testing.]
I ran tests for a full day, both with with version 3.10.0
and your commit https://github.com/kquick/Thespian/commit/a99fbf92590c470f3b7cec74e448fa5d5aa928c2 and can confirm that I never spotted the issue again with https://github.com/kquick/Thespian/commit/a99fbf92590c470f3b7cec74e448fa5d5aa928c2. Thanks Kevin for implementing the fix so quickly!
Environment
uname -a
): Darwin io 18.7.0 Darwin Kernel Version 18.7.0TCPTransport
(on a single machine)Error
When running an actor system on a single machine we get the following stack trace after a while:
In the error log we also see:
I saw that the code in
TCPTransport
has handling for invalid file descriptors but it expectsOSError
to be raised.I can reproduce this in our application when an actor sends large log messages so I wonder whether it is related. I also managed to capture a complete debug log output of Thespian (~ 13MB expanded; the error is towards the end of the log file).
The error is pretty well reproducible but the reproduction scenario is a bit involved (it requires running an application that is somewhat complex to setup) so I'm not sure whether it is helpful to describe it here. I am of course happy to replicate the steps here, provide further details and / or participate in testing but I wanted to get the issue out here first.