search

timeplus-io / proton

A streaming SQL engine, a fast and lightweight alternative to ksqlDB and Apache Flink, 🚀 powered by ClickHouse.
https://timeplus.com
Apache License 2.0
1.38k stars 53 forks source link

Fan-out support for the same source #288

Open yl-lisen opened 8 months ago

yl-lisen commented 8 months ago

The multiplexer fans out one streaming store reader to different streaming queries. This has efficiency of disk read / TFF deserialization, memory allocation etc. But we may introduce new problems like one slow query processing pipeline will slow down other pipelines. So it maintains some metrics for channel and detach the channel and its the pipeline when slowness is detected. 

                                                            -> StreamingStoreSourceChannel -> QueryProcessing Pipeline
StreamingStore Partition -> StreamingStoreSourceMultiplexer -> StreamingStoreSourceChannel -> QueryProcessing Pipeline
                                                            -> StreamingStoreSourceChannel -> QueryProcessing Pipeline

Multiplexer is multiple-thread safe, and a multiplexer is bound to a shard

jovezhong commented 7 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 7 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 6 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 6 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 6 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 5 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 5 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 4 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 4 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.

jovezhong commented 3 months ago

(Jove Github Bot) assuming it is not done, deferred this ticket to the next sprint.