search

rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins
https://www.rabbitmq.com/
Other
11.97k stars 3.9k forks source link

MQTT v5.0 support #2554

Closed wrsrap closed 1 year ago

wrsrap commented 5 years ago

Are there plans to support MQTT v5.0 in this plugin？Thank you.

michaelklishin commented 5 years ago

There are plans but we offer no ETA promises. The earliest release it can happen in will be 3.9.0.

tsturzl commented 4 years ago

I'm willing to contribute here as I'm coming up on needing the ability to use the Shared Subscription feature included in v5.0. If there's any general guidance you can give to how I could begin an implementation of this feature I'd be happy to make a PR hopeful in the near future.

ajayb0507 commented 4 years ago

Hi, Is this still on schedule ? Thanks.

michaelklishin commented 4 years ago

It is still scheduled to be implemented but we offer no ETA as there are too many higher priority items on our roadmap. This is open source software so those who need MQTT v5.0 support can contribute it, or at the very list document all the differences from 3.1.1 which would greatly help with assessing the amount of work it would take.

An0nymous0 commented 4 years ago

look forward to

petersilva commented 3 years ago

in case it is helpful, the feature that most interests me is shared queues. with AMQP 0.9 you just have multiple processes connect to the same queue to do sharing. with MQTTv3, only one consumer at a time. there is a concept of shared queues that promises to address that. Very interested in that feature in particular.

I think the idea is $share/__group_name__/topic_tree ...

where __group_name__ would seem to map to a queue name (in the sense that it accomplishes the same level of grouping a multiple subscribers using a common queue) in AMQP.

michaelklishin commented 3 years ago

Shared queues with repeatable reads will be provided by streams in 3.9, not via MQTT but any future such features likely will lean on them.

petersilva commented 3 years ago

No need for new features on the rabbit side. The existing sharing of queues (has been there last ten years or so) is fine... I just have multiple subscribers connect to the same queue with a prefetch, and I can have 40 subscribers share a high speed queue (that does not build up.) I don't see a way to do that with MQTTv3. I think v5 might help.

adiroiban commented 3 years ago

For the reference, a document with differences between v3.1.1 and v5 - https://github.com/mqtt/mqtt.org/wiki/Differences-between-3.1.1-and-5.0

briggySmalls commented 3 years ago

For me a big appeal of v5.0 is the "Reason code on all ACKs" (in same link as @adiroiban).

If a client attempts to publish/subscribe to a topic that an ACL denies them, the MQTT 3.x specs silently accept the operation. MQTT 5.x introduces the return code for "Not authorised".

petersilva commented 3 years ago

What would be really cool for v5 support: -- map each topic exchange to the root of mqtt topic tree. -- prepend with $share/queue_name/exchange/ .... rest of the topics.

If the exchange is not a topic exchange, it does not participate in the mapping. That way, people can connect to the same broker with either protocol and access the entire broker.

example:

I want five processes to use the same queue "worker" bind to "canada.public" topic on the "xwarnings" exchange in AMQP sharing the messages among the five. That should be equivalent to subscribing to "$share/worker/xwarnings/canada/public" in MQTT plugin.

petersilva commented 3 years ago

fwiw... I use this mapping here: https://github.com/MetPX/sarracenia/tree/v03_wip/sarracenia/moth

There are MQTT and AMQP sub-classes of the "Messages Organized by Topic Header" I use mosquitto for the MQTT stuff, because the rabbitmq isn't rich enough currently.

BastianMattes commented 2 years ago

Any updates on when MQTT 5 will be supported? Do you have a roadmap somewhere on planned features?

lukebakken commented 2 years ago

Anyone asking about MQTT 5.0 support should read @michaelklishin 's comment (link):

It is still scheduled to be implemented but we offer no ETA as there are too many higher priority items on our roadmap. This is open source software so those who need MQTT v5.0 support can contribute it, or at the very least document all the differences from 3.1.1 which would greatly help with assessing the amount of work it would take.

Note that there is useful work that people could do that would speed the process up yet we have not yet received assistance from the community.

BastianMattes commented 2 years ago

This reads like this project is understaffed. That's important to know if you need to decide between different MQTT brokers. Thanks for the honest answer, @lukebakken !

yaronp68 commented 2 years ago

@BastianMattes MQTT 5 support is not a high priority right now. However, we do consider bridging MQTT brokers withRabbitMQ so you will have a solution that has an MQTT layer and Streams / AMQP queues layer facing the IoT processing applications. Would a connector like this (with orchestration) be something you would like to try?

BastianMattes commented 2 years ago

@yaronp68 Thank you for asking, but we don't plan to have any AMQP clients, so if we were to use another MQTT broker, there would be no need for us to have a bridge to RabbitMQ since we wouldn't need RabbitMQ anymore at all.

tsturzl commented 2 years ago

@BastianMattes You're not paying for a support license, and there's not enough interest in this feature. AFAIK you can pay for commercial support for RabbitMQ: https://tanzu.vmware.com/rabbitmq

The idea that RabbitMQ is run by a skeleton crew is kind of silly. The reality is you're either banking on the free time of contributors to implement a niche feature, or you're hoping a company will implement it, but you're not paying that company. Such is the nature of opensource. If this had more interest it might get done faster, but as far as message queues that also support MQTT they basically all only support v3.1.1. You can go pay for HiveMQ or use the community version of VerneMQ if you're really really sure MQTTv5 actually provides all the features you need.

BastianMattes commented 2 years ago

Thanks for the suggestion, @tsturzl , we will consider this in the future.

couling commented 1 year ago

For those of us not familiar with the project, is there a comment on why this issue was closed besides being open for a long time with little progress?

Has it now been deemed that rabbitmq will never support MQTT5?

lukebakken commented 1 year ago

is there a comment on why this issue was closed besides being open for a long time with little progress?

This issue is not closed. I'm not sure why you think that.

Has it now been deemed that rabbitmq will never support MQTT5?

No. RabbitMQ will support MQTT 5 at some point in the future. Hopefully it will ship with version 4.0

michaelklishin commented 1 year ago

Before RabbitMQ supports MQTTv5 we decied to significantly optimize the MQTTv3.1 implementation we have. Chances are, MQTTv5 can ship around the 3.12 time frame, and hopefully it will be in the open source edition (the core team cannot guarantee that).

lukebakken commented 1 year ago

5895 is a prerequisite for MQTT 5.0 support.

lukebakken commented 1 year ago

@t03apt @feixuenihaoma @liamdiprose @yyyyeeeeesss @ramlongcat @hiepthai @gogangipark @andrew-kratik @An0nymous0 @lazystone @msallin @siaron @lddhappy @monkalways @niburhsoj @alaendle @fungiboletus @lhoste-bell @zaksnet @jonaskello @phaseOne @maxlein @BastianMattes @paulojoao @nouzun @damikun @Daschi1 @couling @tsturzl @petersilva @briggySmalls @adiroiban @An0nymous0 @ajayb0507

Hello! Everyone who has asked "When will RabbitMQ support MQTT 5" or thumbed-up this issue, PLEASE take the time to test this new feature. It's the least you can do! The following docker image contains the code in PR #7263 -

pivotalrabbitmq/rabbitmq:main-otp-max-bazel
alaendle commented 1 year ago

@lukebakken - Thanks for pinging us; I'm currently testing MQTT5 support. Looks all good to me - connect with userproperties; publish with userproperties; qos 0/1 - maybe one small improvement idea: is it possible to somehow show the connect userproperties in the management UI? I would have expected to see them below "Client properties" grafik

ansd commented 1 year ago

Thank you @alaendle for testing MQTT 5 and your feedback. I like your idea and implemented it in https://github.com/rabbitmq/rabbitmq-server/pull/8681.

couling commented 1 year ago

@lukebakken Is the shovel plugin not working in this image? Our setup for MQTT has an edge/extranet broker with shovel to/from a core broker. For me to stress test this I'd need that working, but I'm just getting Shovel 'Dispatch Instructions' could not connect to source when I try. Note I've basically dropped this in place of an existing instance with the same config, so this shouldn't be subject to firewall issues etc.

ansd commented 1 year ago

When testing MQTT 5, please start using docker image pivotalrabbitmq/rabbitmq:main-otp-max-bazel since PR https://github.com/rabbitmq/rabbitmq-server/pull/7263 was merged into main branch. (I updated Luke's comment above).

@couling the shovel plugin is supposed to work in this image. The plugin is disabled by default, but you can enable it. The error message you pasted does not provide useful information: I added the error reason to the log in https://github.com/rabbitmq/rabbitmq-server/pull/8686 If you think that the MQTT 5 PR broke the shovel plugin (I would be surprised if it did), please provide us reproducible steps including all config files (ideally create a separate GitHub repo with reproduction steps).

michaelklishin commented 1 year ago

@lukebakken Is the shovel plugin not working in this image? Our setup for MQTT has an edge/extranet broker with shovel to/from a core broker. For me to stress test this I'd need that working, but I'm just getting Shovel 'Dispatch Instructions' could not connect to source when I try. Note I've basically dropped this in place of an existing instance with the same config, so this shouldn't be subject to firewall issues etc.

We will not allow this thread to be hijacked to troubleshoot Shovels. Use Troubleshooting Networking, collect logs on both sides, and if TLS is used, Troubleshooting TLS. If you find something you cannot explain, start a new discussion.

couling commented 1 year ago

@michaelklishin apologies, I wasn't looking for general troubleshooting only pointing out that the docker image provided here to test MQTT 5 support doesn't seem to work in a way I can test MQTT5 in a way that's meaningful to me.

alaendle commented 1 year ago

@lukebakken - also the "standard" mqtt library for JS doesn't seem to be compatible with RabbitMQ (from my perspective due to a bug in mqtt.js https://github.com/mqttjs/mqtt-packet/issues/92). However other broker seem to be more reluctant to protocol violation.

alaendle commented 1 year ago

@ansd - And many thanks for implementing my suggestion ❤️ - things now show up as expected: grafik

ansd commented 1 year ago

Thanks for your feedback @alaendle 🎉 https://github.com/rabbitmq/rabbitmq-server/pull/8752 fixes this issue.

ansd commented 1 year ago

MQTT 5.0 blog post: https://blog.rabbitmq.com/posts/2023/07/mqtt5

petersilva commented 1 year ago

Fantastic Progress! I can't use it... as my usage requires shared subscriptions which the announcement says is deferred. I already described a mapping earlier in the thread that should make it straight-forward, as far as I can tell... map shared subscription group names to AMQP queue names, and everything should "just work."

Still pleased to hear of the progress.