@ReubenBond I initially began the port of Orleans.Streams.Kafka in one of my branches and started comparing its implementation to the current Orleans.Streaming.EventHubs logic - EventHubs and Kafka are conceptually very similar, both are streamed pub-sub solutions using partitioned logs as their backing store. After doing some analysis, I want to hit pause and generate a good gameplan before continuing further.
Overview
Upon analysis, there are a number of concepts that exist in the EventHubs project that do not yet exist in the Kafka library:
The Queue Cache (along with all of its complex logic around pressure monitoring, statistics, purging, etc)
Checkpointing
Receiving while considering checkpointing and the queue cache
Proposal
I would like to propose a new library - Orleans.Streaming.PartitionedLogs to serve as a common abstraction between these two concepts.
This would allow a great deal of complex (necessarily so) code sharing across Event Hubs and Kafka, rather than being Event Hubs specific - this also paves the way for future integration use cases that are conceptually similar to Event Hubs and Kafka.
I believe we could carefully bring log from EventHubs -> PartitionedLogs without introducing binary breaking changes, etc.
@ReubenBond I initially began the port of Orleans.Streams.Kafka in one of my branches and started comparing its implementation to the current
Orleans.Streaming.EventHubs
logic - EventHubs and Kafka are conceptually very similar, both are streamed pub-sub solutions using partitioned logs as their backing store. After doing some analysis, I want to hit pause and generate a good gameplan before continuing further.Overview
Upon analysis, there are a number of concepts that exist in the EventHubs project that do not yet exist in the Kafka library:
Proposal
I would like to propose a new library -
Orleans.Streaming.PartitionedLogs
to serve as a common abstraction between these two concepts.This would allow a great deal of complex (necessarily so) code sharing across Event Hubs and Kafka, rather than being Event Hubs specific - this also paves the way for future integration use cases that are conceptually similar to Event Hubs and Kafka.
I believe we could carefully bring log from EventHubs -> PartitionedLogs without introducing binary breaking changes, etc.