How can we scale or balance load on a MQTT client in nodeJs.

[arihantdaga]

I am using Mqtt for connecting IOT devices. In my case i'll be having tens of thousands of devices reporting sensor values etc simultaneously. And my Node Server is responsible for listening to all sensor and other action topics and interact with database accordingly. I am assuming that If it is via HTTP i can scale it easily by putting load balancers and running multiple instance of the Node Application. But in case of MQTT if i want to scale it, or if i want to distribute load to different machines, i could not find a solution. Is there something i can do for handling huge amount of messages using MQTT.js ?.

[RangerMauve]

You should consider feeding your MQTT events into a different message queue that supports round-robin balancing like RabbitMQ, then have a bunch of Node worker processes consuming from that queue.

[amanthegreatone]

@RangerMauve can you please elaborate a little more on this tip. I am also trying to build an IoT application and facing issues with node event loop getting stuck when consuming data from the MQTT client at a very high frequency. I am trying to work with node js child processes but seem to have some disconnect in trying to do this load balancing.

[RangerMauve]

@amanthegreatone You should look into the concept of "work queues" or "task queues". The idea is that you have a central broker that receives all events, and then one or more workers that connect to the broker. Instead of all workers getting all events, the broker will spread the events across the workers so that the load is balanced between them.

What you can do in this case is feed all your MQTT events into a RabbitMQ broker without processing them, then have workers connect to RabbitMQ to have the load spread between them.

Here's a tutorial from their docs which might help get you started.

Even though I suggested RabbitMQ, there are loads of alternatives that might suite your use-case or environment better. Zero MQ is one alternative.

To spin up multiple workers easily, take a look at the cluster module, and various libraries out there for simplifying its use.

The problem with using MQTT for handling huge numbers of messages, is it's not made for having tonnes of messages on a single topic that a single consumer uses. A better approach is to have different topic patterns and having different consumers set up per-pattern.

There may be work out there on using MQTT brokers with load balancing, but from what I recall there's nothing in the actual spec that would facilitate this.

[arihantdaga]

@amanthegreatone I think most brokers are now supporting $queue and $shared topics... In my case Emq Broker is supporting $queue topics. So i can run multiple instances of the node application subscribing to the $queue topics.in theier docs it's mentioned here . Hope it helps. Although last time i checked, This library wasnt supporting $shared subscription, but in my case i tried with queue susbsciption and it was working fine.

[efcjunior]

Hi @RangerMauve Do you think that concept of "work queues" or "task queues" continues being a good option even with MQTT 5?

Thanks!

[RangerMauve]

Hi @RangerMauve Do you think that concept of "work queues" or "task queues" continues being a good option even with MQTT 5?

Thanks!

I haven't adopted MQTT5 in any projects yet, but the new features don't take away from what was there already. You might need to do a bit of work to load balance work between clients if you have a lot of tasks going through an individual topic.

[github-actions[bot]]

This is an automated message to let you know that this issue has gone 365 days without any activity. In order to ensure that we work on issues that still matter, this issue will be closed in 14 days.

If this issue is still important, you can simply comment with a "bump" to keep it open.

Thank you for your contribution.

[github-actions[bot]]

This issue was automatically closed due to inactivity.