Let a service be a subscriber to a topic

[jsabater]

As per the discussion on this topic on the forums, please implement the ability for a service to be a subscriber to a topic.

[dsuch]

Hello @jsabater,

I am starting to work on it now - if you have had any additional ideas or wishes in the meantime, please let me know so they can be included.

Regards.

[jsabater]

As an addition to what's already been put into that forum post, at the moment I can only think of, maybe, the ability to have multiple consumers of queue jobs or messages, same as producers. You know, so that you can process them faster, whenever that's possible. But most probably you already took that into consideration. :-)

I'll let you know if something else comes to my mind. Thanks!

[dsuch]

If multiple services subscribe to the same topic then all of them will receive the same message - because al of them will be subscribers.

Is that what you meant or did you mean multiple services subscribing to the same topic and only one of them will receive each message in a (say) round-robin fashion?

[jsabater]

The latter, round-robin fashion. But now that you mention it, I think it would be best for Zato to support both modes so that multiple uses can apply. What do you think?

[dsuch]

What I am working on right now is adding a new kind of endpoints, services, just like there are already WSX, REST, AMQP and the rest.

Round-robin is a very low-level change that, once implemented, will work for all kind of endpoints / subscribers, and this is a bigger task for another time.

In the case of services and round-robin - do you think this would be really practical?

I can imagine multiple REST endpoints receiving messages one at a time because they would be, for instance, multiple instances of the same application.

But with services? Do you see a need for different services, with different business functionality, listening for messages from the same business topic?

[jsabater]

In my opinion, both models make sense in different scenarios:

1. All consumers get a copy of the message. It makes a lot of sense as you may have to communicate the same message, whatever it means or contains, to a number of endpoints which are different systems that manage to live together trough the ESB. For instance, some system does something and puts a message in the queue. Then two systems consume a copy each and update their data in different manners but due to the same action that happened elsewhere.

2. Multiple consumers of the same queue. Common scenario on any application that needs to scale. Classic example in the tourism industry a user making a reservation. This booking is stored in the database, triggering a number of events through messages in queues. The OK is returned quickly to the user in the screen while a number of operations are being processed in the background by consumers of those messages. Two cases would be somehow updating availability or similar information and sending a confirmation email. This last case, if having to scale up the application, could mean having a consumer of compose-new-reservation-email job types and storing them somewhere while also adding a send-email type of job on some other queue. Finally, another consumer would send emails, of this type or any other type. Any of these consumers could be duplicated for better performance.

[dsuch]

You are right and there is no doubt that both are useful but I was wondering specifically about multiple Zato services receiving messages in a round-robin fashion.

As long as there is not any kind of one Zato service = one external system setup in a given architecture, I just struggle to find a use-case for it.

[dsuch]

Having given it some more thought - I am sure a use-case could be found. At any rate, round-robin for services will be added too once I get to adding that logic so this was really about my wondering aloud.

Another reason why round-robin is useful is that I would like for Zato to become an MQTT broker in one of the future releases and given that we have pub/sub already, this should be just a matter of implementing the protocol-level specifications.

[jsabater]

Just re-read my last comment and I am not sure whether I expressed myself properly. Nonetheless, my point in the second scenario was that multiple producers may add messages to one or multiple queues, and multiple consumers may pull messages from a single queue to process those messages faster.

The example being:

• Producers: A new reservation in the system requires an email to be composed and sent.
• Consumers: A new email composition has been required. The more consumers, the faster they'll go, but we don't want to duplicate the job, so it's one job per consumer, first-come, first-served.

Please let me know if further clarification is needed.

[dsuch]

Everything is clear, @jsabater - this is also how I see it work.

I was just drawing a distinction between the cases of consumers being Zato services vs. external systems because I was thinking in the context of my current works.

But other than that, I see we are in total agreement.

[dsuch]

This is done - here is a blog article about it. This does not include everything we talked about but the rest will be implemented separately.

https://zato.io/blog/posts/pubsub-service.html