search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.54k stars 1.3k forks source link

Multi-Stream Consumption #7514

Open sajjad-moradi opened 3 years ago

sajjad-moradi commented 3 years ago

Currently one realtime Pinot table only consumes from one stream. For use cases in which multiple existing streams are the input source for one Pinot table, the workaround is to use stream processors to read from different streams, perform necessary transformations, and write the output of all transformed events into another stream. Then one realtime pinot table can be set up to consume from that output stream. This requires maintaining two systems: the stream processor jobs and also pinot realtime tables. It would be nice to add capabilities to pinot realtime tables to consume from multiple topics.

sajjad-moradi commented 3 years ago

I'm working on a design doc for this and will share it here soon.

icefury71 commented 2 years ago

@sajjad-moradi this is a great idea. Lemme know if you had any update on this issue.