Ever considered use Apache Kafka as Event Store

[wli3]

Instead of plain SQL service.

it is

• ordered It is easy to rebuild readmodels
• distributed No bottle neck of scaling event store
• Publish-Subscribe model Catch up map to workers

[brettmorien]

Hi William.

My team is attempting to build a CQRS system using Kafka for an event store (in Node, so this library is more useful as a guide to us). It is showing promise, and you're right that it's especially good for projections, but there are a few gotchas to watch for. The major issue is we haven't found a way to do arbitrary queries of events for a given aggregate, which makes reconstructing aggregates on-demand difficult. We are attempting a redis-based distributed cache of aggregates, so we'll see how that goes.

SQL does a really good job in this regard, plus having a reliable globally incrementing identifier is super helpful.

[wli3]

Hi Brett, Glad to hear that you are trying for this approach. I am still in learning Kafka phrase.

You are right, seems Kafka have partition about 10-40 in mind. While any aggregate could be a partition, in this way, thousands of them.

SQL does have a decent job and actually no problem at all in production. However, SQL could eventually be a bottle neck for scale out.

And speaking building CQRS in Node, since you are using dynamic language, you can have way more fancy metaprogramming ;)

[brettmorien]

Fancy metaprogramming is appealing, but we're already back-filling in capabilities you get out of a statically-typed system out of the box :)

Our use of Kafka is tending toward topics being per aggregate-type, and using aggregate Ids as the partition keys to keep the events near each other.

A Sql database on good hardware will take a long time to become a bottleneck, especially with this append-only access pattern and few indexes.