Kappa+ architecture support - Githubissues

bvt123 / SCH

GNU General Public License v3.0

1 stars 4 forks source link

Kappa+ architecture support #9

Open bvt123 opened 4 months ago

bvt123 commented 4 months ago

Several types of Jobs:

Realtime (MV on Insert)
Batch (periodic runs over Stage tables)
Periodic runs over Source tables with full day/month reload
Combined
- run real-time MV
- store to Stage
- replay the previous day after midnight, or the previous month on 1st day of the next month (to fix mistakes)

Clickhouse is a better store for Stage data than Kafka https://www.youtube.com/watch?v=ExU7fJFw4Bg