Topic prefix stripping in server-to-server connections

smurfix commented 9 months ago

Hi,

While FlashMQ supports prefixing topics on messages it sends and receives, it doesn't seem to be able to remove a prefix from incoming or outgoing topics, at least as far as flashmq.conf manpage is concerned.

This would afford transparently tunneling arbitrary messages through a shared server.

Mosquitto can do it ;-)

halfgaar commented 9 months ago

I can see why you would want it. I'd have to spend some time looking at how complex it would be to make, and what the performance implications are (I tend to avoid adding overly heavy features).

smurfix commented 9 months ago

I hear you. A "strip the first N components of the incoming / outgoing message" feature would work for most if not all users, if that's easier than a comparison; I didn't yet encounter any such link that needed more complex arrangements.

In any case, on the positive side, at least this feature has no performance impact if it's not used.

marcelrv commented 8 months ago

This topic mapping would indeed be very appreciated. @smurfix you indicate topic prefixing when sending, how did you accomplish that as that would already do most of the job in my case, but did not recognise how to do that from the conf file nor from the bridge code

smurfix commented 8 months ago

I currently accomplish topic prefixing with RabbitMQ, but I'd rather switch to something faster + more extensible (FlashMQ has this nice plugin architecture …) if at all possible.

halfgaar commented 8 months ago

I did some checking for feasibility. One question pops to mind: if there is authentication in place, do you expect that to happen before or after the mangling? Let's say we're talking about messages on topic a/b/c:

In incoming publish: do you want the 'write' permission to be checked on prefix/a/b/c or a/b/c?
In delivering a publish to a subscriber, same question: do you want that 'read' permission to be checked on prefix/a/b/c or a/b/c?

There are some architectural difficulties if you would want the ACL check to happen with the prefix. Logically, I'd say without the prefix makes most sense.

smurfix commented 8 months ago

As long as you're consistent whether the checks are done on the paths as seen by the client or as seen by the server (= every other client that doesn't have mangling applied to it) I can write an ACL to match.

Thus, my answer is "whatever happens to be easier to implement".

wiebeytec commented 8 months ago

OK, then I'll give it a go. Prefix stripping on receive is easy. Adding a prefix on send is a bit more difficult, but if we can do it at the very last step, meaning after authentication on the original topic string, it's probably doable.

Do note that it will incur some extra overhead, because there are certain optimizations it can't do then.

Probably the next feature you want is multiple bridge connections with load balancing. It's on my list...

halfgaar commented 8 months ago

I created a branch with a test version: https://github.com/halfgaar/FlashMQ/tree/PREVIEW-bridge-prefixes. If compiling is too hard, I can probably give you a .deb package.

The bridge config sections supports a local_prefix and remote_prefix option. It should act the same as Mosquitto's, in terms of how it subscribes and what it adds and removes.

The prefix options are defined per bridge, not per pattern. There are some conceptual problems with having them per pattern. I'm not sure how Mosquitto does it...

The ACL checking turned out to be the following:

Remote prefix is removed and local prefixes is added before ACL write checks. So, the ACL write checking is done after the mangling.
The read ACL checks (delivery to subscribers) is done before mangling. So, the ACL is done on the path as it appears locally. Then, after ACL, the local prefix is removed and the remote prefix added.

halfgaar commented 8 months ago

Actually, I was a bit premature. There are various issues with that branch still.

smurfix commented 8 months ago

Thanks. Looking forward to testing it once you've got the issues ironed out.

Building a .deb is not a problem.

smurfix commented 8 months ago

OK, then I'll give it a go.

👍🏼

Probably the next feature you want is multiple bridge connections with load balancing. It's on my list...

Heh. Not really. My volume is high but not that high.

What I do want is redundancy, i.e. connect my MQTT servers in an arbitrary mesh and lose no messages (*) if any one of them hangs / fails / whatever. I'm currently using my own protocol on top of MQTT for this and I'd love to be able to bypass the whole thing.

(*) except exact duplicates of course; when a sensor tells me five times that it's 18° out there I don't care how many of those arrive – as long as the number is greater than zero.

halfgaar commented 8 months ago

As an update; I was doing some prep work for this change, which kind of opened up can of worms of things that needed fixing. So, I'm working on a new release, coming in a few days I hope, and then I can continue with this.

smurfix commented 8 months ago

Thanks for the update. Not a problem.

halfgaar commented 7 months ago

I think I'm close to giving you a .deb to test. For what OS+version do you want it?

smurfix commented 7 months ago

Cool. Standard Debian stable should be fine.

halfgaar commented 7 months ago

You can download a test version for Bookwork here. It's a release build, so optimized and no symbols. At this point, I don't expect test scenarios I need debuggers for, just whether it behaves correctly. I can always make a debug build if required.

Install with dpkg -i.

This comment still applies as to how it works.

The more scenarios tested, the better. Like bridges with persistent sessions (non-clean start and non-0 expiry), severed connections, queued messages, using aliases, etc.

halfgaar commented 7 months ago

I'm eager to any test results, both positive and negative alike :)

smurfix commented 7 months ago

Yeah and I'm eager for some "free" time in which to test the setup I have in mind …

I'll try to shovel some hours free next week.

halfgaar commented 6 months ago

Has the shoveling worked ;)?

smurfix commented 6 months ago

On 28.05.24 08:27, Wiebe Cazemier wrote:

Has the shoveling worked ;)?

Still shoveling, unfortunately.

Best laid plans and all that grumble.

-- -- regards -- -- Matthias Urlichs

halfgaar commented 5 months ago

I created new test builds, based off 1.15.2:

There were some architectural changes in master, so this branch was adapted to that.

smurfix commented 3 weeks ago

Could you rebase this to the subscription-id branch? That'd let me set up a mostly-integrated test environment which I hopefully can (lightly) torture over the holidays.

halfgaar commented 3 weeks ago

It has some merge conflicts, and more with the subscription IDs. I can, but I'd rather first finish the subscription IDs and have that in master.

smurfix commented 3 weeks ago

Thanks. No hurry.

halfgaar commented 3 weeks ago

I tried to rebase, but now the tests fail. I don't know why yet.

I also saw some likely mistakes. So, this still needs some work.

halfgaar commented 17 hours ago

I have a new test branch:

https://github.com/halfgaar/FlashMQ/tree/WIP-bridge-prefixes

The prefix is not applied automatically to the bridge's subscribe and publish directives. It felt too magic to do so, plus it would mean the prefixes are applied to all subscribe and publish directives, which is not good.

Let me know how it works out.

halfgaar / FlashMQ

Topic prefix stripping in server-to-server connections #81