search

robinhood / faust

Python Stream Processing
Other
6.72k stars 535 forks source link

Topology Plan #470

Open anismiles opened 4 years ago

anismiles commented 4 years ago

Export Faust logical plan or topology

Is there any plan to export Kafka Streams like topology for Faust App logical plan? Something like this: 

Topology
Sub-topologies:
Sub-topology: 0
    Source:  KSTREAM-SOURCE-0000000000 (topics: [conversation-meta])
    --> KSTREAM-TRANSFORM-0000000001
    Processor: KSTREAM-TRANSFORM-0000000001 (stores: [conversation-meta-state])
    --> KSTREAM-KEY-SELECT-0000000002
    <-- KSTREAM-SOURCE-0000000000
    Processor: KSTREAM-KEY-SELECT-0000000002 (stores: [])
    --> KSTREAM-FILTER-0000000005
    <-- KSTREAM-TRANSFORM-0000000001
    Processor: KSTREAM-FILTER-0000000005 (stores: [])
    --> KSTREAM-SINK-0000000004
    <-- KSTREAM-KEY-SELECT-0000000002
    Sink: KSTREAM-SINK-0000000004 (topic: count-resolved-repartition)
    <-- KSTREAM-FILTER-0000000005
Sub-topology: 1
    Source: KSTREAM-SOURCE-0000000006 (topics: [count-resolved-repartition])
    --> KSTREAM-AGGREGATE-0000000003
    Processor: KSTREAM-AGGREGATE-0000000003 (stores: [count-resolved])
    --> KTABLE-TOSTREAM-0000000007
    <-- KSTREAM-SOURCE-0000000006
    Processor: KTABLE-TOSTREAM-0000000007 (stores: [])
    --> KSTREAM-SINK-0000000008
    <-- KSTREAM-AGGREGATE-0000000003
    Sink: KSTREAM-SINK-0000000008 (topic: streams-count-resolved)
    <-- KTABLE-TOSTREAM-0000000007
ask commented 4 years ago

We went a different route from Kafka streams, instead of generating a topology tree in advance we just have async generators that subscribe to topics and tables as they are used.

We do have the graph: http://localhost:6066/graph/ graph

This graph is populated as components in the app are starting. We could make the topology output above from that, but currently the tables are not connected to agents. To connect tables to agents we would have to add it to the graph at runtime as the table is used.