search

Yuppiechef / cqrs-server

An opinionated Clojure CQRS/ES implementation using Onyx, Datomic, DynamoDB, Kafka and Zookeeper.
MIT License
211 stars 13 forks source link

Why store in DynamoDB when you have Kafka? #7

Open greywolve opened 9 years ago

greywolve commented 9 years ago

Hey Deon,

Really love this idea. I've been reading quite a bit about event sourcing myself, and came across this a few weeks ago. I think this might be a very good fit for Cognician too. Thanks for the writeup and resources!

Just wanted to ask you a quick question regarding your storing of events in DynamoDB. Why do this when Kafka is already persisting them to disk safely? Are you doing this for indexing purposes?

neverfox commented 9 years ago

Doesn't Kafka only persist to some SLA that is usually in the range of days or weeks? While I guess you could configure it to be permanent (I'm not really sure), it doesn't seem to really be suited as a perpetual database. And then there are queries...

jackdempsey commented 8 years ago

@greywolve Neverfox is correct. Kafka is designed to store data on the order of days, not forever. It is also not a typically queryable datastore. Think of it as an awesome pipe to shove things through on their way to somewhere else.

neverfox commented 8 years ago

TBF, Kafka's own documentation says that it is suited for being an ES, if you set the persistence level to "forever". And since I'm not 100% on where queries have a place against the raw ES, it's possible that the OP has a point.

CmdrDats commented 8 years ago

Sorry - A bit late to the party on this one. The primary reason is that storage is dirt cheap, and you don't have to worry about the scale down the line. The basic querying becomes an automatic bonus down the line if you want to rerun a particular subset of events to build and aggregate.

If you can get the same thing with Kafka or have other reason for preferring using only Kafka, fine - I just felt dynamodb was a solid, cheap, long-term fire-and-forget event store solution.