booksbyus / zguide

Learning and Using ØMQ
http://zguide.zeromq.org
Other
3.44k stars 1.65k forks source link

Establishing direct connections through a central broker #833

Open SegfaultCreator opened 3 years ago

SegfaultCreator commented 3 years ago

Hello everyone,

I am currently trying to exchange our token-ring based network communication on a broadcast basis (spread toolkit) with a more efficient approach using Zeromq. We have a distributed architecture which consists of up to 25 separate executables, which run on 10+ computers in a local network. In this architecture, we have some kind of the Dynamic Discovery Problem described in chapter 2 of the tutorials as it is flexible which computer runs which executables. We implemented the XPub/XSub pattern proposed for this, but we noticed that all of the data now runs through the proxy server. What we want to have instead is a central broker that is responsible for establishing direct connections between pubs and subs (like the host of a big party that organizes the event, but the communication happens between the party joiners). However, I did not find any pattern, that fits our use case, in the documentation. I thought about having the broker save all the publishers and subscribers to manage new communication partners. Is there anything I have overseen or are there any major issues that contradict using such an approach?

Thanks in Advance SegfaultCreator (now mutated to MessageDropper)​

rich-coe commented 3 years ago

I've used the following before (on another protocol) and have implemented and using this on zmq.

Central broker uses XSUB/XPUB and listens for and republishes 'HELLO' messages. It may also maintain a list of service endpoints and responds to requests for the service list.

Client connects to central broker subscribing to HELLO messages. Client sends HELLO message contains list (service name, listen socket) it provides. Client ignores HELLO messages sent from self. When client receives HELLO message from other clients, update internal registry of service names, with socket endpoints.

Client responds to HELLO with a OLLEH (or some other message) with it's service list, or the broker can respond to the joining client with the current service list.

When a 'client' wants to send to service name, initiate connection endpoint with service by service name. If expecting async reply, create unique 'reply' service name on local 'client's' listener endpoint. Include REPLY (service name, listen socket) in the protocol message to service.

On zmq there's the 'slow initial connection problem' so you may have establish the pub/sub connection between services during service registration instead of at message initiation.

On the Central broker, when sockets close, remove the service from the list and broadcast GOODBYE with service name(s) to all clients.

sappo commented 3 years ago

Dafka uses the approach you described. Have a look the central tower (https://github.com/zeromq/dafka/blob/master/src/dafka_tower.c) and the beacons used in the peers of the network (https://github.com/zeromq/dafka/blob/master/src/dafka_beacon.c)

SegfaultCreator commented 3 years ago

Thanks for these fast and promising answers. Dafka seems to fit very well. If i may ask @sappo , as you are a maintainer of Dafka as well: Has it been tested on 20.04 LTS? I have some dependencies problems with leveldb upon configuring even though i installed it (apt install libleveldb-dev)

sappo commented 3 years ago

@SegfaultCreator I do my development on 20.04, so it should work!

SegfaultCreator commented 3 years ago

@sappo Great to hear that you have it running under 20.04!
Unfortunately, i still get linking errors with leveldb after trying out clang-11 with lld-11 and gcc with gnu-ld linker. Do you have any special installation of leveldb or did you also use the liblevel-db-dev=1.22-3ubuntu2 provided by ubuntu repository?

sappo commented 3 years ago

No I don't have any special installation. This is what apt says I have installed:

libleveldb-dev/focal,now 1.22-3ubuntu2 amd64 [installed]
libleveldb1d/focal,now 1.22-3ubuntu2 amd64 [installed,automatic]