Tracking: Custom Partition mapping

infinyon / fluvio

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.

https://www.fluvio.io/

Apache License 2.0

3.82k stars 496 forks source link

Open sehz opened 3 years ago

sehz commented 3 years ago

To support chatbot example that @ajhunyady is working on, we need following capability:

Declarative partition mapping allow user to map fields in record to partition key. We should extend CLI/API to

ajhunyady commented 3 years ago

Ideally I should have the ability to name the partitions to match session id:

5282645fae22c226da3d4cbd4953ed89a573355a
09daabf4ddbce1fdc1b7d71ba18226a666d3be5f

sehz commented 3 years ago

Yes, we can add logical id partition and map to storage id

0   <=> 5282645fae22c226da3d4cbd4953ed89a573355a
1   <=> 09daabf4ddbce1fdc1b7d71ba18226a666d3be5f

In the CLI and API, we can provide producer and consumer based on logical partition id instead of storage id

github-actions[bot] commented 3 years ago

Stale issue message