To improve the fault tolerance of stream system, we propose to use short-lived stream instead of long-running stream in production. Short-lived stream are scheduled regularly just like scheduled batch jobs. Every execution of the schedule jobs will compute the stream workflow on a brand new set of resources, which consists of the following three arguments.
Execution = F(fact data, workflow definition, new compute resources)
The derived data and existing compute resource have short lifecycle which only span one execution (or to span two executions for redundancy purpose to avoid single point of failure).
This way simplify the mindset to reason fault tolerance of stream system. You don't need to consider when to backfill, or the burden to monitor and maintain a long-running stream system. Everything will be new in the next execution.
It also separates the production traffic and backfill traffic. The two executions happen in separate resources so no impact to production traffic.
zzfukai
commented
4 years ago
To improve the fault tolerance of stream system, we propose to use short-lived stream instead of long-running stream in production. Short-lived stream are scheduled regularly just like scheduled batch jobs. Every execution of the schedule jobs will compute the stream workflow on a brand new set of resources, which consists of the following three arguments.
The derived data and existing compute resource have short lifecycle which only span one execution (or to span two executions for redundancy purpose to avoid single point of failure).
This way simplify the mindset to reason fault tolerance of stream system. You don't need to consider when to backfill, or the burden to monitor and maintain a long-running stream system. Everything will be new in the next execution.
It also separates the production traffic and backfill traffic. The two executions happen in separate resources so no impact to production traffic.