Pipeline Communication (ZeroMQ / gRPC)

[GerritJa]

Regarding to our discussion at yesterdays meeting, i searched again for comparisons between 0mq and gRPC. Here are some interesting results:

• https://bbengfort.github.io/observations/2017/09/04/message-throughput.html

Here are some further evaluations:

• https://github.com/bbengfort/go-rpc-throughput/blob/master/throughput.ipynb

I guess relevant for us are the grpc and the router evaluation. Although the REQ/REP solution has much higher throughput it's also a synchronous communication and leads to less message overhead due to the stateful connection (see http://zguide.zeromq.org/php:chapter3#Recap-of-Request-Reply-Sockets ).

REQ: ".. The REQ socket sends, to the network, an empty delimiter frame in front of the message data. REQ sockets are synchronous. "

Router: "... It creates identities for its connections, and passes these identities to the caller as a first frame in any received message. "

Here is another discussion on stack overflow about grpc and zeromq:

• https://stackoverflow.com/questions/39350681/grpc-and-zeromq-comparsion

Open for discussion ...

[khudemann]

Yeah I also saw the site's you posted. Additionally I also found those slidesets here: https://de.slideshare.net/pieterh/overview-of-zeromq just a few numbers here. It was a presentation of the CEO of the company developing the zeroMQ library

Here slides from google where they describe all the features of gRPC that could also help argue why we use it now. https://de.slideshare.net/borisovalex/enabling-googley-microservices-with-http2-and-grpc https://de.slideshare.net/VarunTalwar4/grpc-overview

And here another discussion about performance and benchmarking.

I guess now we could argue that we are aware that the performance is slightly lower if we use gRPC but since it provides so many features out of the box it makes sense to use it in terms of overall development time for the first prototype. In the protocol of the meeting it was mentioned we could also use both and compare how big the performance impact(or overhead generated by gRPC) is, which could also be an idea to maybe implement a small benchmarking example to test both against each other. It's quite hard to find reliable information on which of those would fit better for our use case, but since we have to have the first prototype ready in just 3 weeks we should start with gRPC and see how it goes. In parallel we could also have a look at zeroMQ, which would be also interesting for me to compare the two and see how big the difference in our deployment scenario really is.

[khudemann]

performance data tested against recent master branch by gRPC guys: https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528&widget=1747498211&container=897341821

[Krymnos]

I guess now we could argue that we are aware that the performance is slightly lower if we use gRPC but since it provides so many features out of the box it makes sense to use it in terms of overall development time for the first prototype.

I also think that is a valid reason for now. Anyways in the end the only ultimate factor to decide which system is best for our use case is by testing both systems in our use case ^^

[khudemann]

Haha thats right :D We can also discuss with the others later in the meeting today.