search

iggy-rs / iggy

Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second.
https://iggy.rs
MIT License
1.83k stars 85 forks source link

feat req: exactly once semantics #426

Closed rtc11 closed 8 months ago

spetz commented 8 months ago

Could you please elaborate more? Exactly once delivery is impossible to achieve in a typical system consisting of additional infrastructure than just messaging e.g. databases, caches etc.

Insomniak47 commented 8 months ago

I'd assume they mean it in the same way that they would with Kafka, impotency keys in the log,and the processing semantics associated with that.

spetz commented 8 months ago

Yeah, I thought that it might be related to some sort of transactional API and read-process-write approach, but still, calling it exactly once, is just a pure marketing and very misleading, as most of the applications out there, don't use just messaging like Kafka, but also other parts of infrastructure. Exactly once can be only achieved by having the message deduplication (or idempotent consumers) in place along with the outbox pattern. Nevertheless, we don't plan to have such feature for now, maybe later on in the future releases.

hubcio commented 8 months ago

I'd assume they mean it in the same way that they would with Kafka, impotency keys in the log,and the processing semantics associated with that.

@Insomniak47 does that answer your question? do you mind closing the issue?

rtc11 commented 8 months ago

I agree you can almost achieve the same with idempotency + deduplication, but EOS is one of two reasons for why I use Kafka Streams. The other reason is the use of topology, which makes you mimic curring. It makes the communication between services really robust

spetz commented 8 months ago

Thanks for the clarification, we'll consider what are the requirements to implement read-process-write approach and see if that'd be something manageable in the near future. On the other hand, what is your typical use case for Kafka Streams if you don't mind sharing it?

Insomniak47 commented 8 months ago

@hubcio this wasn't my issue, I was just kicking around after I saw this on reddit :)

rtc11 commented 8 months ago

On the other hand, what is your typical use case for Kafka Streams if you don't mind sharing it?

Usually decoupling a large domain model. E.g, you have some legacy system with an old database. (not this simple in real life, but you get it)

  1. Stream each table into its own kafka topic
  2. Create a new services for a a portion of the domain that is tightly coupled, .e.g payment, account, content,
  3. Repeat until all the necessary functionality is available
  4. Turn off legacy system
  5. Migrate, improve, delegate to teams whatever