search

FlowFuse / flowfuse

Build bespoke, flexible, and resilient manufacturing low-code applications with FlowFuse and Node-RED
https://flowfuse.com
Other
265 stars 63 forks source link

Investigate MQTT Broker options for FF MQTT Service #4423

Open joepavitt opened 2 weeks ago

joepavitt commented 2 weeks ago

Capture thoughts and discussion around the technological options we have in hosting our own MQTT broker(s).

hardillb commented 2 weeks ago

2 possible paths I think

  1. A single shared multi-tenant broker
  2. A broker per team

I will add strengths/weakness

Single broker

Broker per team

Out of Scope

joepavitt commented 2 weeks ago

What's the need for:

Needs dedicated ports per broker (we can group multiple brokers on single hostname, but may need to shard)

If the broker is dedicated to a team? I guess if a single team then has multiple brokers?

hardillb commented 2 weeks ago

What's the need for:

Needs dedicated ports per broker (we can group multiple brokers on single hostname, but may need to shard)

If the broker is dedicated to a team? I guess if a single team then has multiple brokers?

No, assuming a single broker per team, then each team will need at least a dedicated port (potentially up to 3 ports mqtt, mqtts and MQTT over secure websockets) on a internet facing host in able to access their broker. We can not use hostname based routing for this as that only works with HTTP (OK, technically it can be with MQTTS, but only if the clients support SNI)

joepavitt commented 2 weeks ago

So, user-facing, we'd end up with something like: mqtt://app.flowfuse.com:<teamport> Where the teamport would iterate, etc. as we grow in the number of teams/brokers, but the base URL would be the same for all teams on FF?

hardillb commented 2 weeks ago

So, user-facing, we'd end up with something like: mqtt://app.flowfuse.com: Where the teamport would iterate, etc. as we grow in the number of teams/brokers, but the base URL would be the same for all teams on FF?

we'd end up with something like mqtt://broker1.flowfuse.com:<teamport> and we would iterate both broker1 and the teamport and have to keep track and workout how to fill in the gaps when teams are deleted. The changing of the hostnames is the sharding I mentioned.

(there may also be security challenges around reuse and old "devices" (not FF dev-agent) trying to connect)

ppawlowski commented 2 weeks ago

For the single broker (which is my preferred option), please consider clustering and horizontal scalability from the beginning. This table might be useful

hardillb commented 2 weeks ago

Note on number of ports per broker for the broker per team solution. Eclipse Almen (based off an IBM product) supports running both MQTT and MQTT over Websockets on the same port. So this would mean half the required ports (still 2 if offering both secure and none secure ports, but tempted to say secure only)

joepavitt commented 2 weeks ago

but tempted to say secure only

I agree here - any reason why users wouldn't want this?

hardillb commented 2 weeks ago

I agree here - any reason why users wouldn't want this?

In field devices that don't support secure MQTT

hardillb commented 1 week ago

Currently exploring the following for single/clustered shared broker:

HiveMQ & EMQX both appear to support multi-tenant. Calls with HiveMQ setup to explore and will set up with EMQX

Also looking at the following for a broker per customer/team options:

hardillb commented 1 week ago

EMQX https://github.com/emqx/emqx/issues/12528

hardillb commented 1 week ago

HiveMQ, Vernemq & EMQX has ARM64 (hive ARMv8 only) and AMD64 containers

Mosquitto also has 32bit arm/intel and PPC64-LE

Amlen is build it your self containers (but java based)

hardillb commented 1 week ago

No mater which approach we pick (single shared broker vs broker per team) we will need to be able use LoadBlanncer service types in FF Cloud, docs https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html

@ppawlowski Can you review the above, I don't think we'd have a problem but good to have a second set of eyes

ppawlowski commented 1 week ago

No mater which approach we pick (single shared broker vs broker per team) we will need to be able use LoadBlanncer service types in FF Cloud, docs https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html

@ppawlowski Can you review the above, I don't think we'd have a problem but good to have a second set of eyes

I make a short research about load balancing traffic to the MQTT cluster and Layer4 Load Balancer should be selected since MQTT operates on top of TCP. AWS NLB will be natural choice for us. If needed, HAProxy can be used for this purpose as well.

(BTW I found HiveMQ docs very comprehensive)

hardillb commented 1 week ago

I have a working prototype with emqx (still need to resolve topic ACLs) that includes:

emqx's opensource version looks like it may meet our needs to start with and could be bundled into on prem installs

https://docs.emqx.com/en/emqx/latest/getting-started/feature-comparison.html

hardillb commented 1 week ago

EMQX can do simlar acl checking via HTTP request and has a cache of ACL items per client

https://docs.emqx.com/en/emqx/latest/access-control/authz/authz.html#authorization-cache