search

moscajs / mosca

MQTT broker as a module
mosca.io
3.2k stars 508 forks source link

Why PUB/SUB for redis persistence? #402

Closed rahulbhanushali closed 8 years ago

rahulbhanushali commented 8 years ago

Hi mcollina,

Mosca sounds like a good fit our requirements with all the abstractions that its providing. I am trying to configure mosca to work with Aerospike (for persistence). For which, I have been going through the different persistence wrappers in the lib/persistence directory.

While trying to study the implementation, have figured that it uses redis as both storage (for packets) and pub/sub (for subscriptions channel). Whereas for mongo both subscriptions and packets are stored in the collections.

Couldn't understand why would you have different way of implementing same functionality? Also, how is using pub/sub with redis helping?

mcollina commented 8 years ago

The answer is speed, looking up topics for offline is really hard to implement straight on top of Redis without using Lua Scripts, and complex even with them. Looking them up in memory speed things up a lot.

I would recommend you to implement your solution on top of Aedes, and have a look at https://github.com/mcollina/aedes-persistence-redis.

rahulbhanushali commented 8 years ago

So, mosca will be a lot slower when used with mongo than with redis as for mongo, the subscriptions are stored in a collection and the fetched and filtered for a client?

mcollina commented 8 years ago

latency is a bit higher, but the throughput will be the same. In aedes they will both use the same logic.

syberkitten commented 6 years ago

Hi @mcollina Do you think we can use a different transport other then redis for the PUB/SUB ? (ps. I'm speaking of Aedes)

The reasons being that redis cannot be replicated (as of date) to other multi-region clusters preserving the pub-sub reactivity.

currently, only certain commands are replicated but not the pub /sub so if we could separate the pub/sub from persistence, it might be easier to scale the system up.

mcollina commented 6 years ago

To the best of my knowledge, this could not be implemented using Redis 2 in a performant way, which is what this still support. It might be that Redis streams could be used to fix this.