search

nanne007 / notes

see issue lists for further discusses
https://github.com/lerencao/notes/issues
9 stars 2 forks source link

Streaming and Batching #63

Closed nanne007 closed 7 years ago

nanne007 commented 7 years ago

Dataflow Model

https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102

nanne007 commented 7 years ago

Strong consistency is required for exactly-once processing, which is required for correctness, which is a requirement for any system that’s going to have a chance at meeting or exceeding the capabilities of batch systems.

nanne007 commented 7 years ago

Already finished it.

Three Problem Solved:

  1. How to group data of unbound? By Windowing.
  2. When to materialize the result? By Triggering.
  3. How to handle late data ? By Accumulating or Retraction.