search

FlowFuse / flowfuse

Build bespoke, flexible, and resilient manufacturing low-code applications with FlowFuse and Node-RED
https://flowfuse.com
Other
265 stars 63 forks source link

MQTT Service #1350

Open joepavitt opened 1 year ago

joepavitt commented 1 year ago

Description

With many customers transferring over from FRED, we have had questions from customers about an equivalent MQTT service in FlowFuse.

Currently, we do not offer a like-for-like solution, and are re-directing customers to https://www.hivemq.com/mqtt-cloud-broker/ as an alternative. Given the traffic coming to FlowFuse from FRED, we should heavily consider offering a like-for-like MQTT service in FlowFuse.

Requested By (Prospects):

Requested By (Customers):

### Tasks
- [ ] https://github.com/FlowFuse/flowfuse/issues/4423
- [ ] https://github.com/FlowFuse/flowfuse/issues/4498
- [ ] https://github.com/FlowFuse/flowfuse/issues/4433
knolleary commented 1 year ago

See also #738 as a previous Epic that was raised to cover User access to MQTT.

ZJvandeWeg commented 1 year ago

Another FRED customer raised this with me, they went elsewhere and not to FlowForge due to the lack of MQTT

joepavitt commented 1 year ago

Is the MQTT broker all they were using at FRED, or was it also the Node-RED hosting?

If the latter, curious to know where they went...

hardillb commented 3 weeks ago

We need a clear understanding of what we want to offer here, but with the following assumptions:

  1. we want to present a single broker URL (e.g. broker.flowfuse.[com|cloud] ,we've already used mqtt.flowfuse.cloud for the existing broker)
  2. we want full topic separation between teams (also considering the request for cross team project nodes?)
  3. we want customers to be able to control access (username/password) for none node-red devices
  4. stretch goal for 3 (certificate based auth....)

This needs a mutli-tenant broker (one that can partition topic space based on user).

I have an experimental plugin for mosquitto (requires development head, not yet released version of mosquitto) that does partitioning based on username sufix e.g. user1@team1 or user1@team2 (https://github.com/hardillb/mosquitto-multi-tenant). This may work for a prototype and may be for small scale on prem deployments, I would like to evaluate some of the more mature offerings for FlowFuse cloud.

We also would need to work out the authentication plug ins required

joepavitt commented 3 weeks ago

Does this get easier/harder if we choose to spin up a new broker for Team/Application?

hardillb commented 3 weeks ago

We still have the authentication work to do (and more reason to break that out of the core forge app which has been discussed as part of scaling work)

And then it gets more complicated for docker and even more for localfs.

It also potentially wastes a lot of resource running very lightly or unused brokers.

hardillb commented 3 weeks ago

Also if we run multiple brokers we need to run them all on different ports, where as a single broker is one IP address one port (we can't use the existing load balancers for this as it's not HTTP)

hardillb commented 3 weeks ago

Another question, do we know what FRED used to offer?

hardillb commented 3 weeks ago

HiveMQ have something similar to what I'm expecting by the look of things https://docs.hivemq.com/hivemq-cloud/quick-start-guide.html#cloud-serverless-quick-start

joepavitt commented 3 weeks ago

Another question, do we know what FRED used to offer?

Not sure, @ZJvandeWeg was that ever shared with us?

ZJvandeWeg commented 3 weeks ago

It was a shared broker, much like the one we offer right now on FlowFuse between Node-RED instances but than you could add additional clients.

MichaelBDavis commented 6 days ago

Two additional prospects: https://app-eu1.hubspot.com/contacts/26586079/record/0-1/4381151 https://app-eu1.hubspot.com/contacts/26586079/record/0-1/3724425202

hardillb commented 5 days ago

Notes from internal discussion: https://docs.google.com/document/d/12BueZ6zpUDAgW3bFa2wVB1dgdWeOsCPh4-aL9INj9Rw

joepavitt commented 1 day ago

Given the progress in #4423 and the technical PoCs built by Ben, we've made the technical decisions required, detailed in the Google Doc linked above.

As such, we now need to move this issue into a Product phase of designing the scope of the offering, pricing, etc. I'll open relevant tasks in order to track the progress on those tasks.

joepavitt commented 1 day ago

Phased roll out of the development effort is likely to be broken into:

  1. Users can provision clients/credentials on the FF Broker, and manually configure MQTT nodes inside Node-RED to utilise these credentials
  2. User can provision clients/credentials, and these are automatically updated and available inside Node-RED Editors associated to the Team.