search

dustin / mqttd

a complete mqtt v5 broker
BSD 3-Clause "New" or "Revised" License
16 stars 4 forks source link

any thoughts on how you'd make this scale outable? #7

Open gilesbradshaw opened 1 year ago

gilesbradshaw commented 1 year ago

just wondering..

dustin commented 1 year ago

it depends on what you mean by this. I have generally run multiple instances bridged together with mqtt-bridge. They're not exact mirrors, but I do replicate portions bidirectionally. If you need to service a lot of subscribers, you can just replicate the keyspace to read-only mirrors and let the internet hit it.

I've not personally run into a situation where a single instance can't run my traffic. I think there's a lot of optimization possible on a single node before worrying about partitioning.

Horizontal scaling could work with key partitioning. The protocol supports server redirection at the connection level, which helps a bit for connection balancing, but not so much key space. For high write / low read rates, we could do a similar horizontal + replicate split with replicas and bridges.

For high read and write rates in a flat space, there could be a challenge. I've not encountered that challenge yet, though.

gilesbradshaw commented 1 year ago

My thought is you could probably do it by making clusters of servers subscribe to each other using MQTT

dustin commented 1 year ago

Right, I think you can do that today with mqtt-bridge. I've not actually tried it N:N lately, but the idea is that every incoming message can get sent to N other servers, but it subscribes such that anything that the bridge sends is never returned. So if server A receives a message from a client, the bridge will pick it up and send it to B and C which will send to subscribers downstream, but not back to the bridge.

I built the bridge before I ever considered writing a broker because I was using mosquitto at the time, and mosquitto didn't bridge mqtt v5 properly and I really needed that to work. Turns out, there are a ton of advantages to running a separate bridge from the brokers, but it might very well do this.