search

aceld / kis-flow

Stream processing framework based on Golang.
https://github.com/aceld/kis-flow/wiki
MIT License
300 stars 24 forks source link

Support DAG glow #1

Open hanxuanliang opened 4 months ago

hanxuanliang commented 4 months ago

Will the kis-flow support DAG scheduling later?

pipeline:
    - depend_on: flow1

like the above config

aceld commented 4 months ago

The current Kis-flow does not yet support DAG (Directed Acyclic Graph) scheduling, but we can perform parallel streaming operations.

For example, in cases like combining flow1 and flow2 to generate flow3:

We can have FunctionS and FunctionL nodes, where FunctionS acts as a storage node and FunctionL serves as a read/write node. These nodes can link the two flows together. Alternatively, we can have flow1's result sent to a message queue, and then flow2 can use this as a data source, initiating another flow to execute.

For configurations like:

pipeline:
    - depend_on: flow1
like the above config

This configuration format is not yet supported. However, it's a very good suggestion. The approaches mentioned for Kis-flow can be adapted to a configuration-based approach.

hanxuanliang commented 3 months ago

Maybe DAG is a little more common. It may also be necessary to introduce the docker runtime environment