search

dotnet / orleans

Cloud Native application framework for .NET
https://docs.microsoft.com/dotnet/orleans
MIT License
10.06k stars 2.03k forks source link

[Question] Orleans Streams future? #6776

Closed cosmintiru closed 2 years ago

cosmintiru commented 4 years ago

Hello all,

I've been digging in the Orleans streams source code for the past two weekends, trying to understand why we are experiencing some weird exceptions using the EventHub provider (I kind of figured it out, also based on some issues already opened).

But, while I was browsing that, there are a couple of things I noticed:

  1. Orleans Streams requirements scream for composition. A possible approach would be to use monadic composition (just like IQueryable does). This should allow us to easily build a pipeline which can be extended with explicit functionalities based on backplane, whilst making it a lot more readable.

  2. The 2nd thing I was thinking of, would Akka.NET be a candidate for the computational unit that handles streams? We're currently using a combination of Orleans and Akka, in which Orleans handles every actor-related stuff, while we use Akka for their concurrency primitives. (I hope I didn't commit some sort of blasphemy 😄 😄 )

I'm not expanding on these subjects because I don't know what are the mid to long term plans, but if you're interested, I can expand and maybe build some PoC.

benjaminpetit commented 3 years ago

Trying to understand why we are experiencing some weird exceptions using the EventHub provider

What kind of issue are you experiencing?

  1. I agree, but do you have an example in mind?
  2. I don't think I know enough Akka.NET do really have an opinion
cosmintiru commented 3 years ago

Hello @benjaminpetit

Thanks for your reply.

Currently we are experiencing message loss when in high load. A couple of days I found out about the slow consumers based on one of the issues opened here, which pointed me to a unit test that which specify how we can configure it. We're about to retest everything and see how it behaves. Also, I was hoping that because of the fact that we are using EventHub, to be able to rewind a lot longer back in time, not only during the span of the cache. We're also trying to segregate the eventhubs, and move the problematic publishers in a separate stream provider with separate agents. Also, in various conditions, it seems that some consumers, simply stop (or we cannot explain why we don't receive messages anymore after a burst).

  1. Regarding the composition, we are successfully using a free monad approach (in other domains), which is basically a custom implementation of a container, like IQueryable. LINQ is a DSL, so think of a set of functions around streaming which allows you to compose in the same way we do with LINQ, with the difference that the DSL is pure and the implementations are attached at runtime, which makes every piece of functionality PnP. Yesterday I've started to work on a PoC and I promise I will come back with a code example in a couple of days and discuss it.

  2. Akka.NET, I suppose you already know that is the other major actor framework in the .NET space. Orleans provides a virtual actor model, while the Akka.NET provides a supervisor actor model. The way I see is that for every streaming agent (in the Orleans space) is a supervisor actor, which watches over all the streams (child actors) and every stream watches over their consumers (using supervision strategy).

In terms of dispatching, they are using the mailbox pattern, with various routing choices ,while the mailboxes and the dispatchers are also customizable. Having this types of primitives makes it a lot easier to instrument. Another aspect that I would be very keen on having, would be that each actor can choose if it should use persistence or not, which by default is event sourced. Which means, that we can basically make every stream, regardless of the backbone, rewindable, and have that configured at the consumer level, not at the stream provider, by simply allowing the developer to attach a table storage to a stream provider, then flip the persistence on or off. (https://getakka.net/articles/persistence/event-sourcing.html). From my point of view, this can be an alternative to running infinite tasks for each consumers and manually maintaining a cursor over a cache.

ghost commented 2 years ago

We are marking this issue as stale due to the lack of activity in the past six months. If there is no further activity within two weeks, this issue will be closed. You can always create a new issue based on the guidelines provided in our pinned announcement.

ghost commented 2 years ago

This issue has been marked stale for the past 30 and is being closed due to lack of activity.