Feature Requests: dequeue multiple and group queues

[nileio]

Hi Mike, Thanks for keeping your node always updated .. I use it on couple of hobby projects and it is very useful. Couple of features that I really would like to see (if I have the hang of the code happy to contrib) is

1. trigger multiple messages with one control command, that is sending a trigger 2 for example will dequeue 2 messages, and sending trigger 5 will dequeue 5 messages.
2. internal multiple queues , that is an array of queues ! grouped by a key such as topic. So messages actually end up in different internal arrays. This is I think a massive design change, but it will be really nice to have a dynamic queue , the way I want it. For example, in my project I want to be able to process 5 messages in parallel at the same time but wait for those 5 to be processed before dropping them. So first I want to be able to release 5 at a time (obviously I can do that with a little loop in a function or so unless we include feature 1), but most importantly I need to group them and release one from each queue at one time. I can't use FIFO approach, and also I dont know in advance how many queues would I need (otherwise I would have possibly created multiple nodes in the flow). This feature might not be a great idea but in all cases just thought to share..

thanks

[drmibell]

I'm glad you find the node useful. The use case you describe makes sense and does not have an obvious solution, at least when the number of queues cannot be determined in advance. I have two thoughts.

First, dequeueing multiple messages is probably best handled by using additional nodes to input multiple copies of the trigger command. Since the user can define an arbitrary triggerCmd string, separating the command segment from the numerical (message count) part could get tricky and end up forcing the user to follow special rules to avoid errors.

Having multiple queues (for example, by topic) is not that difficult in principle, but it might not fit gracefully into the design of the queue-gate node. Preserving the gating function while adding this feature would raise issues such as whether an open, close, toggle or default command should apply to all queues or only selected ones. There are a number of other difficulties, as well. If it would meet your needs, I would be willing to work on a new node based on the queue-gate that would do queueing only and provide some independent control of each queue. Please let me know what you think.

[nileio]

@drmibell thanks for response. Makes sense to me. Yes the multiple queues is an area I am currently investigating. It turns out that the Delay node with Rate Limit option could group messages by topic which effectively provide the feature I need. The issue is that the delay is based on a fixed time span in the configuration unlike your node which has a way to control the queue and release of messages (that feature of peek as well is really good idea). It sounds to me the easiest path is to extend the Delay node and add the option to control release of messages with a control message like yours.. Thoughts ? I am happy to contrib code for that one too, it does not sound like a massive extension for a new node..

[drmibell]

I had forgotten that delay has the "by topic" feature. It seems that trigger also has an option ("handling/each") that selects by topic or whatever message property you choose. You might be able to cobble together something with these nodes (unmodified) to do what you want.

A quick look at the code for trigger suggests that it could be the basis for a multiple queue node, and I suspect that's the case for delay also. Still, two things argue in favor of starting from queue-gate. First, it has many features, like peek, drop, and keep newest message, that users have requested. Second, or maybe first for me, is that I am totally familiar with the code and therefore comfortable making major changes. In the next day or two, I will look at delay and trigger to see how they handle the "by property" feature, in case there is a better way than what I have in mind.

If you decide to have a go at one of the core nodes, let me know how you get on. For now, I will work on queue-gate and post here when there is anything to report.

[drmibell]

I have uploaded node-red-contrib-multiple-queue, which is based on the queue-gate node. It supports multiple, independent queues with most of the features of the queueing mode of queue-gate and a few besides. The documentation is not quite complete, and not all the minor features have been completely tested. If you have an opportunity to try it out and report your impressions, I would be very grateful.

[drmibell]

@nileio, I have just pushed version 0.1.1, which fixes bugs and rough edges in the initial commit. I plan to upload to npm shortly, so if you have any feedback it would be good to have it before then. Thanks for your help.

[drmibell]

Version 0.2.0 adds a few features and (more or less) finishes the documentation. I plan to publish to npm tomorrow (US time).

[nileio]

thanks @drmibell for the update . I will get to test soon and let you know. Really awesome work.. inspired me to publish the nodes that are sitting collecting dust for past two years!

[drmibell]

I have updated node-red-contrib-multiple-queue to version 0.2.1 and have had no bug reports. I don't anticipate doing anything more with it, except possibly a scrub of the documentation. I hope you find it useful.