Pooling

[moll]

Hey!

How do you guys go about pooling? I'm saturating a single Node RPC server process with some computations and would like to transparently have clients use different server workers for requests. It's easy to set up a clustered RPC server, but as client connections are long running, they won't be getting the benefit without client-side modifications. That implies some sort of a pool and per-request round batmanning.

What do others do?

[dfellis]

So the JSON-RPC over TCP works best with lots of small, speedy responses. If you have more CPU intensive work or the payload is larger, HTTP works better, because you can put node-cluster over a set of workers that use the HTTP server and round-robin between them on a host, and you can use haproxy or nginx to marshal a collection of servers together in a similar fashion.

You may even want to go with haproxy for a single server and bypass node-cluster (if you're fine with all of the nonstandard listen ports) because HTTP keep-alive can be configured to hold the connection open for the client but close for the servers so you still get load balancing without cross-server tcp handshakes. http://stackoverflow.com/questions/19946436/haproxy-keepalive-connections-not-balanced-evenly

Beyond that, you could try to write the cpu-intensive portion as a C function and use node-ffi's little known "async" syntax to run the cpu intensive stuff in Node's IO threadpool. I haven't done that yet in production, but I have done threading code in Java, so I'm pretty confident it would work ;) and it could solve your problem here, too, if the cpu-intensive portion is amenable to translation into another language.

[moll]

Hey, thanks for the thorough response.

The requests I have vary in speed and aren't really different from each other on the outside. It's just that some of them return a large set and prepping that data blocks other requests. A lot are in the <20ms range, but a few bigger ones may block dozens others for a few secs. That's the blocking I want to avoid by clustering. Also to take advantage of all the cores of a machine.

I thought about jumping over/back to HTTP, but the whole proxy setup and HTTP sounds like unnecessary overhead for me. The TCP way works great, it's just that it could use a little per-request pooling. :-)

What do you reckon? Throw up a socket on the master RPC server process and forward requests to child processes or run a pool on the client side instead?

[dfellis]

Well, you could do this is with the multitransport's childProcess client and server pieces. You spawn a bunch of children and have them run childProcess rpc servers, then your master spawns clients for each along with one TCP server and round-robins requests from the TCP server to the clients - or goes a bit more intelligent with a "least busy" routing logic.

You could then do a similar pattern with many servers by putting another layer on top of that where one server routes to many servers. But you'll need to write said routing software. (Though it sounds like a good OSS project :))

[moll]

I've since written a light-weight replacement JSON-RPC library that presents a DuplexStream interface for piping plain TCP streams to and from (with an intermediary component doing message framing). With a small stream router in between to send those messages to child workers, it's works marvelously. Hope to open-source it this year, too. :-)

[sh1mmer]

@dfellis have you considered lb_pool?

cc @mranney

[mranney]

Another thing you might want to check out is: https://github.com/uber/tchannel