search

AxonFramework / extension-kafka

Axon Framework extension for Kafka integration to publish and handle Event messages.
https://axoniq.io/
Apache License 2.0
67 stars 28 forks source link

Reading from two Kafka topics in an ACL #5

Closed mtelewicz closed 5 years ago

mtelewicz commented 5 years ago

Hi everyone,

So my team is creating a proof-of-concept suite of microservices using Axon Framework. When we started v3.3.5 was the latest version so that's what we're using.

First, a little description. We have two bounded contexts - let's call them "reporting" and "verification." Each context contains one microservice at the moment. Both microservices are running on top of Spring Boot. Since these two microservices need to talk to each other we created an anti-corruption layer (ACL) between them. Each microservice receives commands via Spring Cloud, and each microservice dispatches events via Kafka. The ACL subscribes to the Kafka topic and translates events from one microservice into commands for the other. The ACL uses a Saga class to handle the events. So far this is working just great.

However, we don't really want both microservices putting their events in the same Kafka topic. We'd like each of them to have their own topic. After searching through the Google Group and reading through the source code a bit it seems we can accomplish this using two Tracking Event Processors - one for each topic. So, my first question is - is that correct?

Assuming it is I imagine you'd somehow associate the existing Saga to both Tracking Event Processors? How would you go about configuring these Tracking Event Processors with Spring Boot?

Appreciate the help!

smcvb commented 5 years ago

Sorry for the late reply @mtelewicz. Times are busy at AxonIQ, and sometimes some people have to wait a little longer for a response then we'd like. Regardless, here I am, and I am sure I can provide some background giving the scenario you've described.

As you've rightfully noted, Axon works with Event Processors as 'the mechanical component to provide events to your Event Handling functions', which come in two flavors: the SubscribingEventProcessor and the TrackingEventProcessor. Both implementation require a different Message Source, where we're going to focus on the Message Source of the latter, the TrackingEventProcessor, which is the StreamableMessageSource.

Any one TrackingEventProcessor can only be tied to a single StreamableMessageSource implementation. Thus, as off now, your statement is fully correct, that each Kafka topic will be a different StreamableMessageSource and thus a different TrackingEventProcessor.

We've marked the idea to merge several Message Sources in to one as a missing feature as well, hence why it's being worked on in Axon Framework pull request #1101.

Once this is in, that should ideally proof as a solution for your problem as well. Granted, we're testing the framework feature mainly through Axon Server and the RDBMS store solution we provide, thus some unknown issues might be expected.

Any how, I think this clarifies your current situation and what the future will bring: For now, you will indeed have to set up two distinct TrackingEventProcessors, one for each Kafka topic. In the future, you should be able to merge those into a single StreamableMessageSource which can back a single TrackingEventProcessor instance.

Hope this sheds some light on the situation @mtelewicz. Lastly, I want to point out I will be closing this issue. This is more so a usage question than a feature request, which is our intended focus of the framework repositories. If you are having any other framework usage questions, you can always drop them on the User Group or StackOverflow (with the axon tag).