idea: use a pipe / socket

[benoitc]

I am not sure what are your plan to improve the performance in the lib, but I was thiking that one way to do it would be opening a pipe / socket with the followiing api:

czmq:socket_new/2 create a new Pid + pipe, The pipe is handled in the socket pipe and fill the process inbox while all control operations are sent to the socket.

what do you think about it?

The complexity with that design is that it will require a way to proxy socket operations to the pipe + an eventloop to manage events coming to the socket. (maybe using libuv?)

Such design may probably goes further than the initial intention so I don't know. Let me know.

[gar1t]

This is interesting. I'll ping you in IRC to discuss (we can update notes here).

My approach in general...

We should keep track of performance issues by measuring and confirm that any new facilities are paying off any added complexity.

One of my complaints about erlzmq2 is that the code is so complicated mere mortals can't really contribute and the library suffers for it. I'd like to make sure whatever we do is drop-dead simple, or at least as much as possible.

There might be some improvements we want to make before something like this. E.g. looking at the performance of this thing as messages get bigger, there seems to be a high cost of using term format for both control and payload send/recv messages. I used this because I'm lazy but it does seem to pointlessly hurt performance. This is not mutually exclusive to the pipe approach btw - just something that ought to be be done first.

Will find you and discuss :)

[benoitc]

I agree this should be done carefully. Since you have the benchmarking test already done it owuld be easy to confirm imo.

Another approach would be multiplexing the messages in the port so you could handle more than one message at the same times. The constraint of passing all the messages on 2 pipes will stay but it will improve the parallelism.

What is your idea about removing the term format exactly?

[benoitc]

About the eventloop approach, this can help:

http://funcptr.net/2013/04/20/embedding-zeromq-in-the-libev-event-loop/

also I did similar stuff in python for libuv:

https://github.com/benoitc/uzmq

[gar1t]

Re concurrency - the port is just a pass through to non-blocking operations within the 0MQ/czmq context. Messages are received and buffered by 0MQ -- there's no real bottleneck on the that end.

I don't quite see the advantage of further buffering messages via select, libevevent, libev, etc. -- they just sit happily in the 0MQ socket structure until pulled out via recv_nowait.

The key to this thing is non-blocking operations. If any operations block, the approach you're discussing, where background receives continue to run, would be completely necessary - not just an optimization. This port has that rule though: absolutely no block operations.

Maybe I'm missing something or not understanding your proposal thoroughly.

Regarding the term formatting - just create a home-brewed encoding/decoding scheme between Erlang and the port -- never call term_to_binary to binary_to_term. This is more a matter of when this change lands rather than if. The library should avoid manipulating, scanning, etc. message payload as much as possible.

On Tue, Mar 18, 2014 at 3:59 PM, Benoit Chesneau notifications@github.comwrote:

I agree this should be done carefully. Since you have the benchmarking test already done it owuld be easy to confirm imo.

Another approach would be multiplexing the messages in the port so you could handle more than one message at the same times. The constraint of passing all the messages on 2 pipes will stay but it will improve the parallelism.

What is your idea about removing the term format exactly?

Reply to this email directly or view it on GitHubhttps://github.com/gar1t/erlang-czmq/issues/13#issuecomment-37987256 .