search

RBMHTechnology / eventuate

Global-scale event sourcing and event collaboration with causal consistency (This project is in maintenance mode. Only critical bugs will be fixed, but there is no more feature development.).
http://rbmhtechnology.github.io/eventuate/
Apache License 2.0
709 stars 99 forks source link

Kafka storage backend #206

Open helena opened 8 years ago

helena commented 8 years ago

@krasserm I've started rudimentary work on this and think a separate eventuate-kafka sub-module and its KafkaEventLog etc are preferable - for people not needing Kafka and it's dependencies. While Cassandra is integrated in the eventuate core module, I don't think it wise to put my PR code in there additionally. Many of us using C* also use Kafka but it would cause bloat. Kafka is used in larger systems like I am on but smaller seem to write directly to C*.

Update by @krasserm: this ticket supersedes #59

helena commented 8 years ago

For timing/roadmap purposes, my hope is to have this PR ready in the next month, at least for review cycles, and leverage it at my kafka summit talk in April :)

krasserm commented 8 years ago

@helena :+1: thanks a lot for working on it, really looking forward to see the PR.

Regarding an eventuate-kafka sub-module, we anyway plan to make Eventuate a multi-module project (see #169). So I think it's ok to initially put the Kafka integration into a separate package on the same level as cassandra and leveldb. It will anyway become a separate module when #169 is due (planned for the next 0.6 release). Also take a look at failure handling which has a strong impact on storage plugin implementation.

We already considered using Kafka as storage backend in the past but stopped working on it for reasons explained in #59. I still think the issues described there must be solved before Kafka can be used as storage backend.

helena commented 8 years ago

@krasserm Sounds good :)

helena commented 8 years ago

@krasserm I need to put this off for 2 or so weeks. If you get to it first, great. Otherwise I'll be back at it soon.

krasserm commented 8 years ago

@helena thanks for letting me know. I have other priorities at the moment, so chances are very low I'll do something on this ticket :smiley:. BTW in 2 weeks #169 should be ready I think.

helena commented 8 years ago

Sweet! K finally getting back to this.

krasserm commented 8 years ago

@helena I'm about to revise the Eventuate documentation and also added an FAQ entry How does Apache Kafka relate to Eventuate? in https://github.com/RBMHTechnology/eventuate/commit/62222f8c54798fd3d934ddd9f0224e3448570334 as I've frequently heard that question. Which of the two integration options do you plan to work on? Or can you imagine another one?

krasserm commented 7 years ago

Just found the new Kafka proposal for idempotent producers and transactional messaging. The idempotent producer is exactly what we need for using Kafka as event sourcing storage backend.

odd commented 7 years ago

That's great news!

On 1 Dec 2016, at 07:46, Martin Krasser notifications@github.com wrote:

Just found the new Kafka proposal for idempotent producers and transactional messaging. The idempotent producer is exactly what we need for using Kafka as event sourcing storage backend.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.