Support a message broker

[shurwit]

As our ecosystem grows, we may need to consider additional communication channels between our back-send services. Specifically, message brokers are common tools used for reliable, asynchronous communication in microservice architectures.

Our philosophy until now has been to limit communication between services, however I do not believe this will be a sustainable strategy in the long run if we intend to continue using a microservice-type model. Interdependencies will continue to grow between our services, so in order to avoid inefficiencies, architectural complications, or tendencies towards monolithic services, I believe we should consider the benefits provided by such a system to augment our existing approach where appropriate.

At the moment, my understanding is that message brokers should be used when it is possible for services to communicate asynchronously. For example, if a request is received to create a new resource on one service, this service may notify other services about this new resource without waiting for a response from each one. Message brokers will not be helpful in serving requests that retrieve data across services, or need to immediately provide the result of an action across services. It is worth noting that it may be possible to avoid these types of synchronous communication in many cases with good design relying on asynchronous communication channels such as this.

In the situation described above, message brokers provide the following benefits:

• Asynchronous communication/notification: Message brokers allow communication to occur asynchronously which prevents inefficiencies associated with error handling when real-time communication is not required.
• Loose coupling: Services do not need to know about the existence or functionality provided by other services. They can send messages to the broker that will be delivered to all interested services, and subscribe to receive notifications about messages that they are interested in. This allows us to keep microservices small and focused on their own tasks, and allow us to more easily introduce new services/dependencies.
• Resilience/reliability: If a service becomes unavailable, the message broker will ensure that any messages that were sent while it was offline are delivered once it becomes available again. This will allow the microservices to communicate asynchronously without needing to worry that the messages may be lost due to temporary unavailability creating inconsistent state.
• Scalability: Reducing the tight coupling between services allows each service to be scaled independently to meet real demand. When services are unnecessarily communicating synchronously, this may mean that services are wasting resources waiting on each other, requiring them all to be scaled together.

Examples of cases where a message broker would be helpful:

• 335

... Also instead of making additonal apis we could introduce Kafka or another message broker and integrate all microservices with it - it will be beautiful and no message will be lost even on crash. At the and of the day all messages will be handled by all involved parties.

Example: Core pushed a message in the queue that a user has been updated/deleted and all the rest wll read it at least once or only once - depends on the case. Groups put message in the queue that group membeship request has been accepted/rejected. Notifications BB will read it at some point and will handle it appropriately - e.g send an FCM (This is the same like what we do at the moment with a simple request - but without the guarantee the request will completed successfully)

In this case no party will care the target service is alive in the moment, because once the target service become alive it will read all pending messages in the queue and pass them one by one.

Also my example with Kafka is, because of the advanced cluster strategies in order to support better availability, but in fact most of the brokers do support such stuff.

What do you think?

Originally posted by @mdryankov in https://github.com/rokwire/core-building-block/issues/335#issuecomment-979298287

[mdryankov]

@shurwit, this is great approach. I gave it a try in November with a real Kafka instance and it looked amazing. However there is bunch of configs that put some pressure on configuration complexity - if AWS resolve it then we're good.

One thing is the circular dependency according to the attempt to mix group BB & polls BB functionality and how we could resolve it? Message broker wouldn't be in help.

Also thanks for the pushing further this task! I think it will be in great help for long run and the upcoming complexity