search

josephmachado / beginner_de_project_stream

Simple stream processing pipeline
https://www.startdataengineering.com/post/data-engineering-project-for-beginners-stream-edition/
91 stars 27 forks source link

Enable exactly one delivery #10

Open josephmachado opened 1 year ago

josephmachado commented 1 year ago

Recommendation from Linkedin

add Flink checkpoints, savepoints, state recovery, and fault tolerance.
We can achieve end-end Exactly once semantics using flink checkpointing and sink transactional semantics combined with Kafka idempotent producer.