(no issue) inspiration / similar

[soenkekluth]

Hi @cliffhall

this is no issue - just wondered if you knew

https://github.com/socketcluster/socketcluster/ https://github.com/primus/primus https://github.com/feathersjs/feathers

[cliffhall]

All good stuff for sure, Soenke. This was just a proof of concept with no dependencies other than socket.io thaat shows how easy it is to do real-time multiserver chat in Node without a database for tracking connected users.

The core concept of mesh topology without centralized oversight could easily be applied to IoT, drone swarms, or any number of problem domains.

[fas3r]

Hello @cliffhall ,

instead of connecting each "server" to each other to keep sync, it would not be better to use redis adapter and use pub/sub to keep everything synced ?

Thanks

[cliffhall]

Hi @fas3r,

From the the associated article to this project:

A few years back, I was consulting for a client who had a Node-based instant messaging feature which had worked pretty well at first. But the number of users grew, and then it didn’t.

Even though they had a number of load-balanced server instances running, they found that the Redis database they were using to track connected users was a big bottleneck. No matter how many instances they ran, they all had to talk to that database.

Now arguably, Redis supports replication and just the previous year clustering support had been added, so that problem could’ve been worked. But the team lead wondered if there might be a way to eliminate the database altogether.

This proof of concept was merely my answer to that challenge.

You're describing a centralized, Hub and Spoke model which has the advantage of being able to control the data easily at the hub, but the disadvantages of the hub getting overwhelmed as traffic increases. If the hub should go down all nodes are affected, and scaling is more complicated to implement and configure.

This POC implements a decentralized, Peer-toPeer model, akin to blockchain, where all nodes have all the data. It is extremely scalable and simply configured. One huge advantage is that the loss of a given node doesn't affect the system at all - there is no core node that can take the system down. A disadvantage is an elevated amount of communications node-to-node.

So when you ask...

instead of connecting each "server" to each other to keep sync, it would not be better to use redis adapter and use pub/sub to keep everything synced ?

... you're really asking "Isn't centralized network topology better than decentralized?"

The answer to which depends entirely upon your situation. You use the architecture that works best for you given the realities of your project. That's why both of these topologies exist.

[fas3r]

Hello @cliffhall ,

Thanks for the details.

I'm (was I guess :+1: ) looking for a client / server ( clusters ) to sync micro services ( clients ) when those start / update / stop.

Removing the redis dependencies is a great idea .

Last question, In the TODO you talk about cluster. You mean module cluster and fork, something like this ?

or using worker_threads, something like this ?

I took the time the look at the code anyway, it's a really good idea.

Thanks.

[cliffhall]

@fas3r yes, the TODO was in regard to Node.js clustering. If you can make a single port on a box be an entry to the cluster that makes it easy for the client to connect deterministically. With this current configuration the client would need to know the possible ports to connect to and choose one at random (the demo let’s you pick the port). So it’s difficult to load balance.

Without external an external load balancing mechanism, we could still add it to this server’s protocol, though, since all server instances know how many clients are connected to their peers. If you connect to a client and it has more than a certain percentage more connections than its least loaded peer, it could send back a message telling you to connect to that peer, then close the connection. The client would then simply connect to the peer it was redirected to.