search

dotnet / orleans

Cloud Native application framework for .NET
https://docs.microsoft.com/dotnet/orleans
MIT License
10.03k stars 2.02k forks source link

Orleans.Streaming.Kafka - Analysis & Plan #8494

Open cmeyertons opened 1 year ago

cmeyertons commented 1 year ago

@ReubenBond I initially began the port of Orleans.Streams.Kafka in one of my branches and started comparing its implementation to the current Orleans.Streaming.EventHubs logic - EventHubs and Kafka are conceptually very similar, both are streamed pub-sub solutions using partitioned logs as their backing store. After doing some analysis, I want to hit pause and generate a good gameplan before continuing further.

Overview

Upon analysis, there are a number of concepts that exist in the EventHubs project that do not yet exist in the Kafka library:

  1. The Queue Cache (along with all of its complex logic around pressure monitoring, statistics, purging, etc)
  2. Checkpointing
  3. Receiving while considering checkpointing and the queue cache

Proposal

I would like to propose a new library - Orleans.Streaming.PartitionedLogs to serve as a common abstraction between these two concepts.

This would allow a great deal of complex (necessarily so) code sharing across Event Hubs and Kafka, rather than being Event Hubs specific - this also paves the way for future integration use cases that are conceptually similar to Event Hubs and Kafka.

I believe we could carefully bring log from EventHubs -> PartitionedLogs without introducing binary breaking changes, etc.

ReubenBond commented 1 year ago

cc @benjaminpetit

cmeyertons commented 1 year ago

@benjaminpetit @ReubenBond thoughts?

Project / assembly name: Orleans.Streaming.PartitionedLogs

High level analysis for files under Orleans.Streaming.EventHubs/Prodivers/Streams

No Event Hub deps

Minority Azure Event Hub specific code

Event Hub specific