search

galatea-associates / fuse-starter-kafka-streams

MIT License
0 stars 3 forks source link

GStream subscribeUpdates operation #33

Open GalateaWade opened 3 years ago

GalateaWade commented 3 years ago

new operation for GStreams that allows joining of multiple streams together without sharing the same key - Can be used to join data streams and ref data streams, allowing output of updates from either stream.

GalateaWade commented 3 years ago

Keyspace K: data stream input Keyspace KR: ref stream input

Use a combination of transformers in the keyspaces K and KR to handle "subscribing" data keys to ref key updates. First time a data key is received, a message will be sent to the KR keyspace to "subscribe" the data key to ref key updates. When ref data is updated, the KR transformer will send a message to the K keyspace with the updated ref data.

DS - Data Stream (K Keyspace) DS_Tr - Data Stream Transformer (K Keyspace) LatestRef1 - Latest Ref Store 1 (K Keyspace) LatestRef2 - Latest Ref Store 2 (K Keyspace) SubReq - Subscription Requester (K Keyspace) Repart_KR - Repartition to KR keyspace (KR Keyspace) SubHandlr - Subscription Handler (KR Keyspace) SubStore - Active subscription store (KR Keyspace) RefHandlr - Handle Ref data updates (KR Keyspace) Repart_K - Repartition to K keyspace (K Keyspace) Joiner - Data Joiner (K Keyspace) Output - output stream (K Keyspace)

Single ref stream, Initial occurrence of data input key Diagram

2 Ref streams, initial occurrence of data input key Diagram