need to implement sendmsg and recvmsg

[kwpatrick]

I've been looking into this. Here's my current take.

• We'll need to be able to send/receive the control information for some of the protocols. That should be straightforward.
• Sending via scatter buffers is a great thing, since practically speaking we'll usually be firing off serialization results, which are usually written to a stream first. Buffering that stream would then be another memory copy, which can likely be avoided. My idea thus far is to write to a buffer pool of unmanaged pages, and then hand these scattered pages to nanomsg. The bad news here is that right now nanomsg is just copying the scatter contents into a unified buffer. He really shouldn't do this, since I know Windows and suspect Linux should be able to send directly from scatter buffers. Hopefully that's temporary, and it will have a path to WSASend in future versions.
• Receiving into gather buffers is interesting, but not obviously useful. At present letting nanomsg allocate and return the message buffer is probably simplest, and should be fast enough if he implements his own buffer pooling (last I saw it was just malloc/free for every message). If he never has internal pooling, then the gather buffers could let us implement our own, but only in scenarios with a well-established maximum message size (since it truncates anything beyond the size of receive buffers).

Once the streaming send and receive are done, I'll try and put together some protocol buffer demos.

[mikerussellnz]

Will this allow the return path to be passed along when using Req / Rep sockets. i.e. through a load balancer to a set of worker threads?

[kwpatrick]

The message control information could be used for that, but honestly it'd make more sense to me to maintain an array parallel to your list of socket file descriptors (that are passed to nn_poll) containing callback information, and update the two lists atomically.

I couldn't say more without more visibility into the threading/balancing scenario you have in mind, though. Suffice it to say that the intention of the change is to allow more efficient sends of serialized data structures, as the use of scatter buffers avoids several forms of overallocation.

Let us know if you see other opportunities though!

[mikerussellnz]

I was thinking a multithreaded server where there is one TCP socket accepting requests and passing them through via inproc to multiple worker threads. I am new to nanomsg and where I got stuck was I get the request from the TCP socket, send it via inproc to one of the workers, the worker sends a reply but then how do I send it to the correct client? I only have one listener socket.

[kwpatrick]

I'd recommend having a look at the following:

Req/rep example of fair queueing https://github.com/nanomsg/nanomsg/blob/master/tests/reqrep.c

Req/rep topology design http://250bpm.com/blog:14

You'll have multiple rep sockets on the same address. Each of them will handle a single request at a time, and thus a reply sent on them will be to the correct client. So each worker will need to receive some sort of context of the reply socket that they should be sending on.

I'd suggest taking this to its own issue if you feel this functionality isn't correctly or conveniently provided, so that it can be properly addressed. I'm actually closing this one, as it was handled by the earlier change set.