intercommunication channel

[dvv]

Hi!

Please, consider adding a duplex way for workers to broadcast simple JSON messages. It may be achieved via a shared unix socket, like it's done at https://github.com/dvv/simple/blob/master/src/server.coffee#L194 for master and at https://github.com/dvv/simple/blob/master/src/server.coffee#L77 for workers.

I just discovered cluster and want to switch to it and to cease maintaining own solution, but I need intercommunication so that workers can report to each other changes in their state. Having redis just for pubsub seems overhead. Another pubsub solutions are either immature or too complex to employ.

TIA, --Vladimir++

[tj]

I already have this. They do not communicate withe each-other though, only worker -> master and master -> worker

[dvv]

I see. What about a broadcaster? So that workers wouldn't need to know of siblings if they wanted to publish a message to every sibling. Consider exchange of ids of connected clients, so that to robustly notify of changes all clients connected to the cluster, not only those connected to a particular worker.

How can I find you at IRC may be?

[tj]

I'm always on #node.js. I dont want cluster to become an IPC library though, that is not my intention. The IPC within the library is for uses within it only, the workers should not be communicating with each-other unless they use a different means of IPC not related to cluster. What is your use-case for this functionality?

[tj]

one thing I would possibly be ok with, is exposing (and updating) an array of pids to each worker so that you do not have to read the pidfiles

[dvv]

sure. pidfiles are outdated technique (imho).

I updated the source, you might want to relook.

I tried to explain the use case at "Consdier exchange ...". I admit the description is poor, along with english i write. Still, the use case is cheap pubsub (no redis, no nothing but a single unix socket); broadcasting of part of worker internal state -- all workers should have list of all connected clients if we want robust comet ootb; memoization of sessions which elimiates the need to access DB on each client request; easy use of storages which otherwise cannot be simply put in multi-core pattern (consider excellent nstore which suffers from being single-core due to file lock) and so on...

I hope we can continue on discussing, as multi-core IPC problem is clearly becoming a stopper nowdays

[tj]

well the pidfiles are necessary for the cli.

I dont disagree, but that is still not a goal for cluster. You could easily still use some third-party library with a local socket for IPC, cluster should not be in charge of, or get in the way of that. Although I could not find a good third-party JSON-based IPC lib either, might be something worth working on (but I believe dnode handles this) dunno

[dvv]

To me, cli is a poorman repl here. BTW, cli (I run node test.js --status) doesn't exit until explicitly CTRL+C is hit. So behaves other cli flavors. Can't track down the issue.

They all way heavyweight for the needs. I outlined the lack of easy-to-use external tools. You see, I don't try to introduce worker-on-worker dependency, no. I want master to relay data come from a worker to another workers (effective broadcasting), that's all. It could be a plugin in cluster terms.

Plus, these messages might be the basis for IPC pubsub, which per se could be a plugin which takes a hash of channel names valued with handlers for messages arriving to those channels.

[tj]

yeah, and I have a repl too lol, you need a cli for init scripts etc, it's still handy..

if node test.js --status does not exit then you have something else going on in your test.js file that is active in the event loop.

ah, yah I used to to have Master#broadcast() but I was not using it, do git show 6a76ae7, we could add it back in if it was used for a plugin

[stuarts]

Was Master#broadcast() added back in as a plugin? I would second the use case for this, especially for realtime applications where client messages could be consumed by a worker -> stored in master -> and broadcast to clients through workers. That seems like a pretty natural extension of maintaining database connections in the master process.

[tj]

yeah but again that is not what cluster is for, if you want IPC, just setup some IPC, relying on cluster's would mean I would have to make it public API, which is just not a goal for cluster

[dvv]

But setting external IPC involves additional degrees of complexity -- another socket to connect to to maintain -- and will be in essense a rewrite/copy-paste of cluster IPC. Please, consider making a plugin, be it under BIG FAT WARNING.

[dvv]

master: broadcast any fully received message to all workers worker: process.notify(msg) to notify master, and process.on('ipc', function(msg){...}) to process broadcast messages from master