Closed andatt closed 5 years ago
The problem you are encountering is that you have a bottleneck on the QuaternaryActors, but no protection against shutting down TertiaryActors before they have completed the asynchronous send to the QuaternaryActor troupe.
There are generally going to be 10 SecondaryActors, and each of those will be creating a TertiaryActor troupe with 10 each, so that will be about 100 TertiaryActors, all trying to send to just 10 QuaternaryActors (just one troupe instance because it is a global troupe). This means that many of the asynchronous self.send()
operations from the TertiaryActors to the QuaternaryActors will be delayed, waiting for an idle QuaternaryActor. However, there is no troupe_work_in_progress
manipulation in the TertiaryActors, so the extras will be shutdown as soon as the original message was received, killing many of these before they have completed the send to the QuaternaryActor.
One simple way to observe this is to introduce an arbitrary delay in the TertiaryActors during which the troupe_work_in_progress
is true (and before responding with the done message to prevent the SecondaryActor from killing the entire troupe). On my system, this arbitrary delay is enough to allow all 1000 messages to be sent.
@troupe()
class TertiaryActor(ActorTypeDispatcher):
def receiveMsg_dict(self, msg, sender):
self.send(self.createActor(QuaternaryActor, globalName="quaternary"), msg["data"])
self.lastmsg = msg
self.troupe_work_in_progress = True
self.wakeupAfter(3)
def receiveMsg_WakeupMessage(self, wakemsg, sender):
self.troupe_work_in_progress = False
self.send(self.lastmsg["from"], "done!")
Naturally, an arbitrary delay is not a good methodology for production code, but it serves to demonstrate the adjusted behavior in this case.
Thanks yet again Kevin, you are a lifesaver! I assumed that once sent the actor system as a whole took care of it if a recipient actor was busy and the message was undelivered. i.e. I assumed this would be queued and have independent existence of the actor which had sent the message.
I now see this is not the case and have modified my code to set the WIP True and only un set once a message is received from QuaternaryActor that its task is complete. Now I get the number of expected messages in the log. The time delay approach you mentioned also worked as you say though not great except for proving the issue.
Hi Kevin
This is a continuation of issue 43 (https://github.com/kquick/Thespian/issues/43), I managed to reproduce what was an intermittment issue in the test code below.
It basically the same code as we were using in issue 43 (https://gist.github.com/kquick/38a23b58e16f1505720a4a18fece012f) but with more data, the data being dicts instead of ints and a couple of hepler functions to count the output in logfile. It seems the critcal factor causing failure here is just that there are now 1000 messages to process as opposed to 100 in the original issue:
For me it hangs at 332 log messages, there is nothing in /tmp/thespian.log. The actor processes which are showing after the hang:
Is this a system resource issue? I dont get why it would be though, as I would have expected the troupe's to only max out at 100 actors in total which should be well within source resources.
Any ideas?
Thanks
Andrew