search

RBMHTechnology / eventuate

Global-scale event sourcing and event collaboration with causal consistency (This project is in maintenance mode. Only critical bugs will be fixed, but there is no more feature development.).
http://rbmhtechnology.github.io/eventuate/
Apache License 2.0
708 stars 99 forks source link

Kafka storage backend #59

Closed krasserm closed 9 years ago

krasserm commented 9 years ago

Use Apache Kafka as event storage backend. This ticket does not include

krasserm commented 9 years ago

See also this thread on atomic batch writes to Kafka. A related thread is this one with this post giving a hint that batch writes to a single partition are atomic, and another post indicating that the new producer probably won't get a batch API.

krasserm commented 9 years ago

Work on this issue is now suspended in favor of #62. Reason is that event replication semantics can only be guaranteed if the event backend store supports atomic writes of event batches. Even though this might be possible with the old message producer in Kafka, I think it is not reasonable any more to use it when Kafka officially recommends using the new producer.

The new producer however cannot be used (at the moment) for atomically writing user-defined event batches. Idempotent producer could be an alternative here but this feature is not even planned for Kafka 0.9.

Storing an event batch as single Kafka message solves the atomicity problem but requires an additional mapping from sequence numbers to Kafka partition offsets. in the worst case, the size of this mapping is equal to the number of events stored in Kafka. The additional complexity of maintaining such a mapping would outweigh the advantages to using Kafka as event log storage backend IMO.