search

kbastani / event-sourcing-microservices-example

Learn about building microservices with event sourcing using Spring Boot and how to deploy a social network to Kubernetes using Docker Compose or Helm.
http://www.kennybastani.com/
GNU General Public License v3.0
367 stars 172 forks source link

Add a historical event log that can be used to query and re-hydrate Kafka streams from Neo4j using Spring Data #6

Open kbastani opened 5 years ago

kbastani commented 5 years ago

One of the critical problems that often goes unaddressed in a CQRS/ES microservices architecture is "how do I query the history of events?"

Secondly, people often ask how they can query the history of events to re-hydrate a new aggregate service with historical domain data.

While I think that Kafka streams provide a nice abstraction for event sourcing, I do think that it has a few deficiencies when it comes to querying streams. Here, I propose an idea that uses Neo4j to store a linked list that replicates topics from Kafka, and preserves ordering and indexing. For this purpose, I will design a "streams" graph using a new Neo4j database and a Spring Boot application. An example of the graph can be found below.

image

The goal here is to provide easy access to teams who are looking to ingest event data from Neo4j as a collection of new events. Neo4j provides a connector to Kafka[0] which will stream event projections back into a Kafka topic, which can then be used to create an event sourced set of aggregates connected to a user.

[0] https://www.confluent.io/connector/kafka-connect-neo4j/

kbastani commented 1 year ago

Going to implement this with Apache Pinot