forwarding Stored messages to fast for Broker to handle !

[smithtekIOT]

Hi Damo.

Im using the queue node and it is amazing, works very well, even after a power blackout it still holds the stored messages. The problem i found is that my Mqtt broker like many others has a limit on how many messages it can receive in 1s. I for example, stored 24 messages on the queue node 1 minute each. I then activated the connection to the broker. The queue module dumped the messages to quickly to the broker resulting in about 75% of the messages getting lost. I tried this with a second broker and it also happened. the volume of messages was timing the connection out.

To get around this i used a delay node and throttles the limit to 1 msg per second. This worked fantastic. However the issue i then had was the delay node was temporally holding the messages, forwarding them every second. If the internet or power outage happened all them messages then become lost.

So to work around this i then looped back from the delay node to the queue node. if the net drops out it started to load the queue node back up with the remaining messages that didn't make it to the mqtt node.

Problem is, for some reason my broker then rejects messages that have been looped back a second time. the Json format and objects look OK, still my broker doesn't like it.

Is it possible you can add a parameter that allows us to control the speed the queue node forwards the messages. is it possible to also throttle it to say 1 msg per second. If there is a power loss or additional internet loss no data would be lost.

Loads of potential, great work.

Please add me on skype "fdisciple"

Dan

[damoclark]

Hey Dan,

Glad you find the node useful.

My first instinct was to look at alternate ways of integrating existing nodes such as the delay node - you know, the whole UNIX philosophy of composability over monolithic design.

But I see the catch-22 situation - there is always a possibility of messages being lost sitting in the delay queue.

Unfortunately, I don't have the capacity to make enhancements to this node at this time. However, I will gratefully accept PRs if your team wanted to implement it. Rate-limiting code from the existing delay node could be merged into the queue node I'm sure.

Regards, Damien.

[smithtekIOT]

@damoclark

Hi Damo do you have an email i can contact you directly please. ?

Regards

Dan

[rzanetti-cpqd]

Might not work for everyone, but my workaround for this was at https://github.com/damoclark/node-red-contrib-msg-queue/blob/master/queue/ready-queue.js#L98

Simply wrapped the queue release into a setTimeout to postpone it by 100ms - this was enough for the ceiling I was hitting in the MQTT broker. The code now reads

        queue.on('next',function(msg) {
            node.send(msg.job) ;
                        setTimeout(() => {
                                node.log('Releasing queue') ;
                                queue.done() ;
                        }, 100) ;
        }) ;