Does the publishing with zmq bridge support multiple subscribers?

[jamesdillonharvey]

Hi,

I am trying to set up multiple clients subscribing to a single publisher.

Does the zmq bridge support this?

If so how do I configure the mama.properties?

I know this is supported in qpid but that required an extra transport option (reply_url). e.g mama.qpid.transport.sub2.reply_url=amqp://qtdeva06:6666

Thanks

James

[fquinner]

Hi James,

I'll need to have a proper look at this but I originally wrote the 0mq bridge to try and avoid all of the connection management stuff that I had to do in p2p qpid proton (after all, that's usually the sort of thing you expect a middleware to take care of and even in qpid proton, it does take care of it if a broker is involved).

It's been a while since I looked at it though and I'll investigate properly when I get a chance but I always envisioned using a zeromq proxy / forwarder for tcp based pub sub implementations and then for larger fanout, just use a pgm implementation so you don't overburden your publisher for high volume fanouts (#3). Will need to have another look though to see what works best for this sort of deployment. I'm guessing it's just a zeromq forwarder where each node is both a pub and a sub, but I'll need to try it out... which I won't get a chance to do for a few days, but I'll get there :).

Cheers, Frank

[jamesdillonharvey]

Hi Frank,

Thanks for the swift reply.

I did think about using a zmq forwarder but felt it will not support the DQ publisher, for example the stale -> recap/initial etc as two subscribers may be in different states.

Our system is built on tiers that fan out downstream with caching proxy/forwarders on each horizontal level. I was planning on building a caching forwarder/proxy with the publisher/consumer being mama-zmq. In this model I would need the publisher to accept subscriptions to a hand full of downstream subscribers. This was working with the p2p qqid but the performance wasnt good.

I understand the reasons for keeping the zmq bridge clean and for most use cases I see the single subscriber would work. I will put some thought into the overall architecure over the next few days and see if another model would work for our needs.

James

[fquinner]

Hi James,

I still haven't tried it yet, but in theory recap / initial in its current form should allow multiple subscribers to maintain different state through a zeromq forwarder. The reason why is because I don't actually use the ZeroMQ request / reply mechanism for subscriber-driven recovery. Instead, a recovering / new subscriber will subscribe to a UUID based topic, then publish a request for initial / recap out to a known topic which will contain this UUID based topic. The publisher will then 'reply' by publishing out to this UUID based topic which only the recovering client will be subscribed to, therefore you're not messing up state for sibling subscribers.

This approach was taken for a few reasons:

• I was keeping one eye on wanting to use a mechanism which could be adapted relatively unchanged to PGM, therefore I didn't want any publisher to have to store any context about its downstream consumers.
• I remember reading somewhere that ZeroMQ has weird race conditions with mixing pub / sub and request / reply patterns in the same context.
• The approach is already used in the Qpid Proton bridge where a broker is used.

Cheers, Frank

[fquinner]

Hi James,

I got curious... Just tried it and it does work with an XPUB / XSUB proxy which looks like this:

#include "zhelpers.h"

int main (void)
{
    void *context = zmq_ctx_new ();

    // Where all publishers will publish to
    void *frontend = zmq_socket (context, ZMQ_XSUB);
    int rc = zmq_bind (frontend, "tcp://*:4559");
    assert (rc == 0);

    // Where all subscribers will subscribe to
    void *backend = zmq_socket (context, ZMQ_XPUB);
    rc = zmq_bind (backend, "tcp://*:4560");
    assert (rc == 0);

    //  Start the proxy
    zmq_proxy (frontend, backend, NULL);

    //  We never get here...
    zmq_close (frontend);
    zmq_close (backend);
    zmq_ctx_destroy (context);
    return 0;
}

And configuration that looks like this for all nodes:

# Socket to listen for connections on and publish out to
mama.zmq.transport.broker.outgoing_url=tcp://127.0.0.1:4559
# Socket to send all subscription messages to
mama.zmq.transport.broker.incoming_url=tcp://127.0.0.1:4560

Though it needed a small change in the bridge to allow it to work where both publisher and subscriber endpoints are remote (currently assumes publishing side is local bind and subscribing side is remote). This change is now pushed back into master so should be ready for you to try out now. Also verified that it works fine with request / reply between any nodes connected to the proxy without cross-talk.

Cheers, Frank

[jamesdillonharvey]

Hi Frank,

This all sounds great and very modular, I am trying to get it working at the moment.

I have built the zmq broker, pulled and rebuilt the mama-zmq lib and updated the properties to what you have above.

My publisher is not blocking on bind anymore (so the new lib worked) but it doesn't appear to be connecting to the broker either. Likewise the subscriber is sending out the dictionary request but getting nothing back.

I am just adding a capture socket to the zmq_proxy so I can see if any messages make it through.

Which applications are you testing with? I should try it out with those and work form there.

Thanks

James

[jamesdillonharvey]

Its working!

I have multiple subscriptions and for the same ticker and the initials/snapshots are not interfering with each other.

I am not sure why but my zmq_proxy() wasn't allowing connections before, I probably mixed up some libraries.

I will keep you updated on how the testing goes but for now its looking good.

Thanks for all your help.

[fquinner]

Hi James, cleaning up some stuff here and I see that this has been quiet since December so hopefully all is working OK now :)

[fquinner]

Just checking back on this because i made some changes since this was raised - check out the latest default mama.properties for an example of doing fan-out without a proxy in the middle :).