[question] Alternative engines support

[joeblew99]

in the application.go there is a hook into the engine

// engine to use - in memory or redis. engine Engine

I have am using NSQ, as well as NATS. These are both 100% golang based message queues. NSQ has storage.

https://github.com/nsqio/nsq

https://github.com/nats-io/gnatsd

At the moment i use NSQ for microservices and large application. You can string together micro service libraries, without having to get into the HTTP Request Response stuff, but instead just passing messages with body payloads of data between them.

The other thing is that NATS and NSQ dont have great support for interacting with the Web client, Or mobiel client, and so Centrifo would get allot of support if it integrated with these well used MQ libds like NSQ and NATS.

Just an idea at this stage, and i have not had a chance to look at the interface that woudl need to be supported.

Oh and lastly, NSQ and NATS are both in the process of getting a Discovery Service integration, so that they can run on large Kubernetes clusters. Each Server, has a MQ daemon running with it. There are many lookup daemons that know all the MQ daemons, and the topics and channels they all use. So at the code level, you can ask for the topic, and publish to it, and not have to know anything about what or where that other server is.

Basically, in summary, NSQ and NATs, will allow Centrigio to scale out on clouds much easier. I knwo Redis can also scale out on k8 (kubernettes), but its no as decoupled is my point.

[FZambia]

I've seen both and even got some inspiration out of them while writing Centrifugo and Go client for it. Great messaging servers!

There several requirements for engine - storage to keep data (for channel history and channel presence, this should not be necessarily disk storage, keeping data in memory is ok), PUB/SUB mechanism (publishing new messages, publishing internal Centrifugo messages between nodes), expiration mechanism (history and presence data should expire).

For in-memory engine all of these requirements implemented directly in Centrifugo. Redis provides all these features out of the box. Due to its built-in structures it was possible to implement history of fixed size with expiration and presence with expiration.

NATs and NSQ don't fit these requirements well as both of them more like Centrifugo itself while Centrifugo needs something like database with some special requirements above as engine. There are not so many options for engine that can support all Centrifugo requirements.

There is another approach - use NSQ or NATs for PUB/SUB only, use Redis as storage backend. I.e. having engine composed out of 2 different external systems. The benefit is that if you don't need history and presence you can go just with NSQ... But to support such combined engines there must be a mechanism to keep their code outside Centrifugo core and a way to configure it separately. I don't see a good approach how to do this at moment. Maybe running engine as daemon near every Centrifugo node and communicate over protobuffers/flatbuffers? Seems interesting. But also complex and I am not sure is this a good architecture decision.

[joeblew99]

I think I see what you at saying.

What I do is make things like presence a library, with use the mq layer to interact with it. A message queue like nats can go all the way to the client.

But I can see the contradiction.

At the end of the day the real issue is the protocol of the message queue and binary compat. This is why some devs like http based microservices, because htyp is the one true binary compatible interoperable approach.

Mhh makes me wonder about using the new HTTP/2 push mechanisms as a message queue :)

[joeblew99]

Found one :) https://code.google.com/p/httpsqs/

[FZambia]

@joeblew99 I think I missed an idea:(

[joeblew99]

At the moment you can use NSQ and mango to talk web sockets. Not sure about NATS.

It's not sure HTTP/2 push but good enough for now

[FZambia]

Sorry I still can not understand your last thoughts. Could you tell what you mean in another words?

Also what do you mean under HTTP/2 push? As far as I understand HTTP/2 push will be used to push media resources into browsers and it's not related to bidirectional communication between browser and server (what Websockets used for).

[banks]

@joeblew99 yeah http2 push is a confusing thing - it's NOT arbitrary 2 way messaging with browser. See

HTTP 2.0 server push is not a replacement for technologies such as Server-Sent Events (SSE) or WebSocket. Resources delivered via HTTP 2.0 server push are processed by the browser but do not bubble up to the application code - there is no JavaScript API to get notifications for these events.

From https://www.igvita.com/2013/06/12/innovating-with-http-2.0-server-push/

What is even more confusing is that http2 is used in grpc.io and there it supports "streaming" requests where client makes request to server and then gets a long lived stream of replies back.

That is something http supports at a protocol level, but only because the grpc client is choosing to use it that way, In regular web browser that supports http2 they don't have those client features - they only accept push from server to mean "you might also want to load this other http resource".

You could possibly hack that to force loading a javascript "file" with a jsonp style function call in the body to deliver a payload to your app, but I'm not sure that is a good idea really - that is just trying to turn another transport into a websocket when it was never designed for it. Clients don't guarantee to hold http2 connections open for example or reconnect on failure and I'm not sure your app could control the transport enough to do that yourself either. In other words http2 push is not at all the same as websockets and shouldn't be used as such.

FWIW I love the design of both NSQ and NATS and considered building something like centrifugo on top of them. It could work, but it would really need to be a websocket API that mirrors the underlying API of those messaging systems since they are opinionated and don't really fit the way centrifugo is designed.

NSQ would also support history while NATS wouldn't without some other central (i.e. shared by all centrifugo frontends) storage of the history. The obvious choice there is redis and that is essentially what centrifugo is already - just would need an adapter to publish direct from an NATS consumer (which is about 50 lines of go - I just did it for Scribe ;) ).

NSQ could support history too but you'd have to have a separate NSQ consumer queue for every active user session connected to centrifugo which would mean as number of users connected increases, the memory needed in NSQ grows a lot - current redis design doesn't have that issue with memory only growing with number of //channels// that are active. Plus centrifugo design for storing message ids and syncing last ID seen etc would all not work and you'd have to rely on NSQ consumer protocol to handle that for you.

I'm not sure NSQ is really designed for having tens of thousands of consumers - it's more designed for inter-service communication where it's assumed that each consumer is a service and hence having a separate queue for it is not a big scaling issue.

What I'm saying is, what you ask for makes a bunch of sense and I'd personally be interested to see what you come up with - but it's essentially a websocket interface for those queue's protocols. And centrifugo is really a different thing to that right now.

What is your use case by the way?

[ghost]

that really clears up some confusion for me.

The use case is a small load application. we have rooms, and users subscribe to them, but only certain users see certain messages (role based i guess).

all clients are web browsers (chrome)

I also need server to server web socket communication. again small load

on the db level i am using rethinkdb. rethink can also be used like a message queue , because it support queries that are called "changefeeds". this means that is any data that the query represents is changes, rethinkdb fires an event to the middle tier. there you need a goroutine listening, and then you fire it up to the browser using web sockets. RethinkDB is pretty cool in this regards.

[banks]

Yeah I looked at RethinkDB recently.

It would be possible I think to make an engine for it. But I'm not sure if that is quite what you want...

An engine for rethinkdb would just be same centrifugo pub sub features - what you want it seems is to integrate notifications from rethinkdb and publish them to centrifugo for delivery.

i.e. you want a custom PUBLISHER that gets messages from rethinkdb and publishes to centrifugo, not an engine which is expected to support the full pub/sub/storage/presence etc. features of centrifugo.

You could make that an engine that just didn't support most features and converted subscribe requests for channels into rethinkdb queries, then published the results...

But seems you probably just want to build a custom app that talks to rethink and then published the results to a stock centrifugo with one of the existing engines.

Make sense? Would be interested to hear if you have other ideas and where you go with this.

[joeblew99]

Hey again, It's nice that your interested in this.

I see your point. It all comes down to who is the message queue. I don't think, but am guessing a little that rethink can be the message queue itself. So I 100% agree with you.

So then all that has to happen is for the application to use rethinkdb for normal durable data storage.

From my various experienced with lots of different databases I very much think you and others will find it an awesome dB. Super easy to manage and scale in a single data centre and multiple data centres. I am using it this way now and its very impressive how easy it is , and yet still performance is great.

Writing queries is also very easy.

[joeblew99]

Ah I see your using redis for durable storage. Sorry I forgot. So yes you can either make another data adapter for rethink, so people can use rethink OR redis.

I might add that the configurabikity of rethink is similar to redis. For example in rethink you can configure a table to use " soft" durability. It will hold it is memery and fkush to disk every few seconds. The perf is then about 20 times higher I recall.

And rethink clusters like butter on toast. It just works

[FZambia]

It's a nice area - integrating Centrifugo with app using RethinkDB. As @banks noted - this is not an engine, but just a layer that publishes messages received from Rethink by application to Centrifugo. In this case there are more entities participate in a game - Rethink, Centrifugo, Redis, application backend... But as result you can get an interesting set of features.

[joeblew99]

My main issue with using centrifogo is that it uses redis. There is nothing wrong with redis. It's more that thinkdb offers what redis has, but I get something that can scale horizontally a across data centres from day one with zero devops hassle.

[FZambia]

I see your point, but at moment I don't think it would be easy to implement all Centrifugo engine needs on top of rethinkdb.

Let's look at our requirements and what rethinkdb currently has:

• storage to keep data (history, presence) - yes
• PUB/SUB mechanism - maybe, it seems this can be implemented of top of changefeeds feature - https://www.rethinkdb.com/docs/publish-subscribe/javascript/.
• expiration mechanism (history and presence data should expire) - not yet https://github.com/rethinkdb/rethinkdb/issues/746

As rethinkdb user please correct me - maybe I am missing something. We can be on the alert though.

[joeblew99]

I would say its possible. I use changefeeds for projects and so allow a table to act as a queue. Point 3 is just middle tier logic

On Sun, 27 Dec 2015, 12:47 Alexandr Emelin notifications@github.com wrote:

I see your point, but at moment I don't think it would be easy to implement all Centrifugo engine needs on top of rethinkdb.

Let's look at our requirements and what rethinkdb currently has:

• storage to keep data (history, presence) - yes
• PUB/SUB mechanism - maybe, it seems this can be implemented of top of changefeeds feature - https://www.rethinkdb.com/docs/publish-subscribe/javascript/.
• expiration mechanism (history and presence data should expire) - not yet rethinkdb/rethinkdb#746 https://github.com/rethinkdb/rethinkdb/issues/746

As rethinkdb user please correct me - maybe I am missing something. We can be on the alert though.

— Reply to this email directly or view it on GitHub https://github.com/centrifugal/centrifugo/issues/45#issuecomment-167403069 .

[banks]

Point 3 (expiring keys/garbage collection( can be implemented in the engine, but it significantly complicates the design to have your app managing garbage collection in a scalable, fault-tolerant and distributed way. Failure to do it right could easily result in consistency issues, memory leaks or poor performance one number of channels scales up regardless of how scalable rethink is.

But yes I think you could build an engine on rethink now if that was a goal. The thing that isn't obvious to me with limited rethinkdb experience is whether it would be efficient to implement centrifugo's current model which is really based closely on assumptions about what is efficient in redis.

In general, I could imagine another model which is much closer to rethinkdb's internal model/design which would be a better fit (I.e. a possibly simplified rethinkdb API exposed over websocket) than building a centrifugo engine. I'd guess such a thing has been built already elsewhere...)

In other words, centrifugo is a fairly generic pubsub message broker application, rethinkdb is a full database with change feeds feature. You can model simple generic pubsub in rethink if that is your goal, but that's not the same as centrifugo being a good fit as a front end to your existing data model/change feeds your app might have in rethinkdb. Does that make sense?

Would love to hear your thoughts though - if I have time I will look more into rethinkdb features to see how well it could model this.

On 27 Dec 2015, at 13:08, joeblew99 notifications@github.com wrote:

I would say its possible. I use changefeeds for projects and so allow a table to act as a queue. Point 3 is just middle tier logic

On Sun, 27 Dec 2015, 12:47 Alexandr Emelin notifications@github.com wrote:

I see your point, but at moment I don't think it would be easy to implement all Centrifugo engine needs on top of rethinkdb.

Let's look at our requirements and what rethinkdb currently has:

• storage to keep data (history, presence) - yes
• PUB/SUB mechanism - maybe, it seems this can be implemented of top of changefeeds feature - https://www.rethinkdb.com/docs/publish-subscribe/javascript/.
• expiration mechanism (history and presence data should expire) - not yet rethinkdb/rethinkdb#746 https://github.com/rethinkdb/rethinkdb/issues/746

As rethinkdb user please correct me - maybe I am missing something. We can be on the alert though.

— Reply to this email directly or view it on GitHub https://github.com/centrifugal/centrifugo/issues/45#issuecomment-167403069 .

— Reply to this email directly or view it on GitHub.

[joeblew99]

Yes I see your point. Redis is pretty efficient. Here's some tips: But with rethink and the sharing the efficiency can be handled due to its inherent sharing. Also check out "soft" transactions in rethinkdb.

I think you will find it can really scale and be managed much easier.

I use rethink as a queue, with simple coroutine's listening and then firing up to clients over web sockets.

Checkout the to do example: https://github.com/frankdugan3/GoRethink_TodoDemo

[FZambia]

Let's summarise what we have in this discussion at this moment.

Not so many databases can offer everything to support all Centrifugo features out of the box. Redis is perfect in this.

NATS and NSQ are more messaging systems. It would be possible though implement PUB/SUB on top of them and other features on top of other databases. Also it could be possible to implement only PUB/SUB for example and let other features like history and presence as not implemented. But in this case with those features left we also loose something that makes Centrifugo unique. So I think it would be possible only if in future we find a way to make engines pluggable. And keep such engines out of core.

If we consider databases capable to support all our features - at moment I see 2 real candidates here: RethinkDB and Tarantool. Tarantool don't have truly PUB/SUB at moment (but something can be implemented on top of lua). Tarantool is fast and flexible, has master-master replication, not so popular in the world yet unfortunately. RethinkDB don't have TTL yet and a bit different in terms what is this and how it works. But both look promising in some aspects. RethinkDB is quite popular (btw I see a lot of hype about Rethink but I don't really know is it used widely now?) and can be a good companion for Centrifugo - as both positioned as tools for real-time web. Also it seems it scales well.

But we need a real use case scenario and users that need alternative engine implementation to promote this further - otherwise we will grow wide but stay at the same place. Lets keep this question open, maybe we can just experiment with something and see how it goes.

[joeblew99]

I am 100% biased in that I prefer a golang based storage layer. Just want to be frank. Maybe boltdb is a good solution ?

On Wed, 10 Feb 2016, 22:32 Alexandr Emelin notifications@github.com wrote:

Let's summarise what we have in this discussion at this moment.

Not so many databases can offer everything to support all Centrifugo features out of the box. Redis is perfect in this.

NATS and NSQ are more messaging systems. It would be possible though implement PUB/SUB on top of them and other features on top of other databases. Also it could be possible to implement only PUB/SUB for example and let other features like history and presence as not implemented. But in this case with those features left we also loose something that makes Centrifugo unique. So I think it would be possible only if in future we find a way to make engines pluggable. And keep such engines out of core.

If we consider databases capable to support all our features - at moment I see 2 real candidates here: RethinkDB and Tarantool. Tarantool don't have truly PUB/SUB at moment (but something can be implemented on top of lua). Tarantool is fast and flexible, has master-master replication, not so popular in the world yet unfortunately. RethinkDB don't have TTL yet and a bit different in terms what is this and how it works. But both look promising in some aspects. RethinkDB is quite popular (btw I see a lot of hype about Rethink but I don't really know is it used widely now?) and can be a good companion for Centrifugo - as both positioned as tools for real-time web. Also it seems it scales well.

But we need a real use case scenario and users that need alternative engine implementation to promote this further - otherwise we will grow wide but stay at the same place.

— Reply to this email directly or view it on GitHub https://github.com/centrifugal/centrifugo/issues/45#issuecomment-182590169 .

[banks]

I see only very limited value in building engines that store data on the centrifugo node.

Centrifugo's entire scaling design really relies on external storage so that you can easily scale number of front end servers. Even if you can easily serve all your load from one node, I don't consider single node to be a viable option for production as it is a single point of failure.

The current in-memory engine is really only there for testing without redis dependency. I don't see any real cases where you would want to use that in production unless you literally don't care about availability of the system. There's no real value having another persistent engine for testing either.

Of course it would be possible to build an engine with BoltDB (say) that managed replication and communication between Centrifugo instances, but then you are effectively building a custom distributed database inside Centrifugo which is almost certainly not a good idea. If you really have a need for that you should probably focus on building a great distributed DB and add a WebSocket API directly...

Now if there was a pure-go embedded distributed storage and communication library that has already done all the hard distributed systems stuff then that could be interesting, but I don't know of one. Closest would be something like Serf which solves communication and membership but not storage or replication (and is eventually consistent so takes some extra reasoning about).

For now I don't see mileage in building custom engines that are so impractical for production. Something like RethinkDB makes sense if people are already using Rethink for their app.

Perhaps if there is demand for niche engine implementations then we should just make them pluggable so people can build their own exotic ones without us distributing them in core Centrifugo binary.

On 10 Feb 2016, at 23:01, jow blew notifications@github.com wrote:

I am 100% biased in that I prefer a golang based storage layer. Just want to be frank. Maybe boltdb is a good solution ?

On Wed, 10 Feb 2016, 22:32 Alexandr Emelin notifications@github.com wrote:

Let's summarise what we have in this discussion at this moment.

Not so many databases can offer everything to support all Centrifugo features out of the box. Redis is perfect in this.

NATS and NSQ are more messaging systems. It would be possible though implement PUB/SUB on top of them and other features on top of other databases. Also it could be possible to implement only PUB/SUB for example and let other features like history and presence as not implemented. But in this case with those features left we also loose something that makes Centrifugo unique. So I think it would be possible only if in future we find a way to make engines pluggable. And keep such engines out of core.

If we consider databases capable to support all our features - at moment I see 2 real candidates here: RethinkDB and Tarantool. Tarantool don't have truly PUB/SUB at moment (but something can be implemented on top of lua). Tarantool is fast and flexible, has master-master replication, not so popular in the world yet unfortunately. RethinkDB don't have TTL yet and a bit different in terms what is this and how it works. But both look promising in some aspects. RethinkDB is quite popular (btw I see a lot of hype about Rethink but I don't really know is it used widely now?) and can be a good companion for Centrifugo - as both positioned as tools for real-time web. Also it seems it scales well.

But we need a real use case scenario and users that need alternative engine implementation to promote this further - otherwise we will grow wide but stay at the same place.

— Reply to this email directly or view it on GitHub https://github.com/centrifugal/centrifugo/issues/45#issuecomment-182590169 .

— Reply to this email directly or view it on GitHub.

[joeblew99]

Vanadium does the distributed store and encryption. It's on github. Have a look.

On Thu, 11 Feb 2016, 03:21 Paul Banks notifications@github.com wrote:

I see only very limited value in building engines that store data on the centrifugo node.

Centrifugo's entire scaling design really relies on external storage so that you can easily scale number of front end servers. Even if you can easily serve all your load from one node, I don't consider single node to be a viable option for production as it is a single point of failure.

The current in-memory engine is really only there for testing without redis dependency. I don't see any real cases where you would want to use that in production unless you literally don't care about availability of the system. There's no real value having another persistent engine for testing either.

Of course it would be possible to build an engine with BoltDB (say) that managed replication and communication between Centrifugo instances, but then you are effectively building a custom distributed database inside Centrifugo which is almost certainly not a good idea. If you really have a need for that you should probably focus on building a great distributed DB and add a WebSocket API directly...

Now if there was a pure-go embedded distributed storage and communication library that has already done all the hard distributed systems stuff then that could be interesting, but I don't know of one. Closest would be something like Serf which solves communication and membership but not storage or replication (and is eventually consistent so takes some extra reasoning about).

For now I don't see mileage in building custom engines that are so impractical for production. Something like RethinkDB makes sense if people are already using Rethink for their app.

Perhaps if there is demand for niche engine implementations then we should just make them pluggable so people can build their own exotic ones without us distributing them in core Centrifugo binary.

On 10 Feb 2016, at 23:01, jow blew notifications@github.com wrote:

I am 100% biased in that I prefer a golang based storage layer. Just want to be frank. Maybe boltdb is a good solution ?

On Wed, 10 Feb 2016, 22:32 Alexandr Emelin notifications@github.com wrote:

Let's summarise what we have in this discussion at this moment.

Not so many databases can offer everything to support all Centrifugo features out of the box. Redis is perfect in this.

NATS and NSQ are more messaging systems. It would be possible though implement PUB/SUB on top of them and other features on top of other databases. Also it could be possible to implement only PUB/SUB for example and let other features like history and presence as not implemented. But in this case with those features left we also loose something that makes Centrifugo unique. So I think it would be possible only if in future we find a way to make engines pluggable. And keep such engines out of core.

If we consider databases capable to support all our features - at moment I see 2 real candidates here: RethinkDB and Tarantool. Tarantool don't have truly PUB/SUB at moment (but something can be implemented on top of lua). Tarantool is fast and flexible, has master-master replication, not so popular in the world yet unfortunately. RethinkDB don't have TTL yet and a bit different in terms what is this and how it works. But both look promising in some aspects. RethinkDB is quite popular (btw I see a lot of hype about Rethink but I don't really know is it used widely now?) and can be a good companion for Centrifugo - as both positioned as tools for real-time web. Also it seems it scales well.

But we need a real use case scenario and users that need alternative engine implementation to promote this further - otherwise we will grow wide but stay at the same place.

— Reply to this email directly or view it on GitHub < https://github.com/centrifugal/centrifugo/issues/45#issuecomment-182590169

.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/centrifugal/centrifugo/issues/45#issuecomment-182674305 .

[banks]

Vanadium is an interesting project. Didn't have time to look in detail yet.

But what makes you actually think it would be a good alternative? Seems it's primary focus is on security in distributed systems where they are not all in the datacenter - i.e. mobile apps and multiple different services in different locations. It seems more like an RPC toolkit with some storage features. It's "peer-peer synchronised storage" is more like offline/online sync between intermittently connected devices not really a real-time replicated data store from what I can see.

Maybe we should flip this conversation around: can you describe what you actually need, and why Redis doesn't meet that need? Then we can find good technical solutions that do. Speculating on random technologies that sound fun and could possibly be pressed into working is really not the way to build a system.

It seems your only criteria so far in the things you suggested were that they are written in Go which seems a very odd constraint considering that some have been external binaries anyway (i.e. you don't need to care about the code). Can you elaborate on why?

FWIW I totally see possible use-cases for all sorts of backends - maybe even Vanadium - but unless we have real people trying to solve real problems that require those solutions, it's really not sensible to try to design them!

[joeblew99]

I don't like redis only because its not golang. Really I am not joking. I typically deploy golang code to mobile, embedded and servers. I don't use openssh even, but instead a golang equivalent. It's makes everything much smoother

On Thu, 11 Feb 2016, 16:48 Paul Banks notifications@github.com wrote:

Vanadium is an interesting project. Didn't have time to look in detail yet.

But what makes you actually think it would be a good alternative? Seems it's primary focus is on security in distributed systems where they are not all in the datacenter - i.e. mobile apps and multiple different services in different locations. It seems more like an RPC toolkit with some storage features. It's "peer-peer synchronised storage" is more like offline/online sync between intermittently connected devices not really a real-time replicated data store from what I can see.

Maybe we should flip this conversation around: can you describe what you actually need, and why Redis doesn't meet that need? Then we can find good technical solutions that do. Speculating on random technologies that sound fun and could possibly be pressed into working is really not the way to build a system.

It seems your only criteria so far in the things you suggested were that they are written in Go which seems a very odd constraint considering that some have been external binaries anyway (i.e. you don't need to care about the code). Can you elaborate on why?

FWIW I totally see possible use-cases for all sorts of backends - maybe even Vanadium - but unless we have real people trying to solve real problems that require those solutions, it's really not sensible to try to design them!

— Reply to this email directly or view it on GitHub https://github.com/centrifugal/centrifugo/issues/45#issuecomment-182925928 .

[banks]

With respect that is a pretty niche rationale. Redis is way more widely supported in package managers, PaaS, tutorials etc. as well as being battle tested etc.

Building a custom distributed database just because you'd rather deploy Go doesn't sound like a recipe for a good system design!

[joeblew99]

:) well depends hat your building.

Anyway. Lets close this issue. Its been hacked to death....

On Thu, Feb 11, 2016 at 6:31 PM Paul Banks notifications@github.com wrote:

With respect that is a pretty niche rationale. Redis is way more widely supported in package managers, PaaS, tutorials etc. as well as being battle tested etc.

Building a custom distributed database just because you'd rather deploy Go doesn't sound like a recipe for a good system design!

— Reply to this email directly or view it on GitHub https://github.com/centrifugal/centrifugo/issues/45#issuecomment-182968105 .

[FZambia]

@joeblew99 let's keep it open so our users know about current state of alternative engine support in Centrifugo - this is an important topic anyway.

[ghost]

OK - cheers

[CodingNemesis]

Can Centrifugo work with LedisDB (http://ledisdb.com/)? Most Redis clients can apparently connect to it.

[FZambia]

@CodingNemesis no unfortunately. There is a work in issue #100 to decouple Centrifugo packages so it would be possible to write custom engines. As LedisDB has no PUB/SUB support then it will be impossible to implement scaling Centrifugo to several machines on top of it. I think we will need some extra code that will allow to connect Centrifugo nodes into cluster to communicate with one another without any third-party database (like Redis at moment). And use this code inside custom engine implementation. Then choosing database to keep temporary data would be much easier.

[banks]

I think we will need some extra code that will allow to connect Centrifugo nodes into cluster to communicate with one another without any third-party provider.

I think we'd need to consider that very carefully. Implementing a distributed message bus inside centrifugo sounds like a big step and one we might quickly regret. The simple thing of full mesh and broadcast to all peers doesn't scale as connections and messages grow quadratic ally with cluster size. It also leaves you with complicated partial failure cases and great difficulty in describing the actual reliability guarantees that are provided.

I'd actually suggest if we were to consider that, that we should make pubsub part a separate engine from storage part so you could write custom providers for that but. For example NATS.io, NSQ, RabbitMQ or Kafka for pubsub with DB X for storage of history etc.

On 3 Oct 2016, at 08:48, Alexandr Emelin notifications@github.com wrote:

@CodingNemesis no unfortunately. There is a work in issue #100 to decouple Centrifugo packages so it would be possible to write custom engines. As LedisDB has no PUB/SUB support then it will be impossible to implement scaling Centrifugo to several machines on top of it. I think we will need some extra code that will allow to connect Centrifugo nodes into cluster to communicate with one another without any third-party provider. And use this code inside custom engine implementation. Then choosing database to keep temporary data would be much easier.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[FZambia]

@banks I agree with risks, I think it will be theoretically possible without separating pub/sub part from engine. Engine still needs to provide what it does now but inside custom implementation sth like NATS could be used. Let's put this another way - I think we could write some code that uses another PUB/SUB providers (like NATS) that users could use in there custom engine implementations. Of course it could be possible to make PUB/SUB part separate - but current Redis implementation shows that we don't need to make it fully separate as Redis provides everything Centrifugo needs.

[FZambia]

Centrifugo v1.6.0 has a pluggable engine mechanism. But I have not found anything better then Redis for engine. I created Nats-based PUB/SUB engine as a proof of concept. It works but lacks our features such as recovery and presence. So anyway sth like Redis required to make it fully functional.

[criloz]

@FZambia love the nats PUB/SUB engine in nats, I will use it, the db part should be implemented in redis at the moment, but I am thinking in experiment other alternatives like serf and raft in centrifuge for my use case,

[FZambia]

@criloz cool! Please tell if you end up with something interesting – and you can always write into our gitter chat

[FZambia]

Closing since most of discussion here not up-to-date.