Serial event processing in a chain

[Horrendus]

Sometimes it could be needed to wait until the next bee in the chain is finished before that bee receives the next event

My use case: Mixcloudbee gets a list of Cloudcasts that I want to download using an ExecBee & youtube-dl. Opening youtube-dl 10 or more times blocks my IP from mixcloud and also uses quite much processing power. So I want to wait until the ExecBee is finished before sending the next event to it.

Current Design Idea would look something like this:

• Serial Event Processing (SEP) Bee gets an Event
• SEPBee checks somehow (internal state on SEPBee or next Bee in Chain) if the next Bee in the chain is currently busy
• If not busy Event is sent to the next Bee
• If busy Event is put into a queue
• Whenever the "finished" Event from the next Bee is retrieved, SEPBee takes an Event from the queue and sends it to the next Bee

Feedback, thoughts?

[internetimagery]

From a usability perspective, would it make sense for the execute hive to take an optional max number of concurrent processes in its own settings configuration?

So you can throttle it to whatever level you desire ( in this case a limit of 1 ), without requiring another separate hive?

[rubiojr]

No idea how the implementation should look like, but this sounds like a really useful addition!

What if we add a serial-exec-bee instead that execs commands serially reading from an in-process queue so we don't have to change beehive internals to support this?

[Horrendus]

Serial Exec would also be possible. But a really good solution would be to (a) either support it in the Beehive internals, so that this could be used by every Bee or (b) make a specialized Bee that would allow to sit in between any 2 Bees and queue up the events and deliver them one by one to another Bee (when that one finishes work). No idea if it is possible, would need to play around with Beehive :)

[rubiojr]

For some additional ideas, added a very basic Redis based alternative for YouTube videos to https://github.com/rubiojr/beehive-youtube-dl, where an external consumer process for a simple Redis queue is used.