Orleans Recipes and Templates

[ElanHasson]

After being in the Gitter chat on and off for a few years, I see variations of a recurring themes: "I want to do X but I am unsure how to accomplish this the Orleans way" or "I want to do X, should I use Y or Z?".

The current docs explain a lot of things, but they don't necessarily tie the functionality together in a cohesive manner. There are samples for this, but for the most part they are not production ready applications and are more usage demonstrations of some of main aspects of the Orleans public API surface.

I believe there to be a set of core use cases that probably cover 80% of usage scenarios in the wild. This is an attempt to capture those scenarios and document how to implement them The Orleans Way.

Templates

One of the great things about Orleans is the flexibility to configure almost every aspect of it. This benefit can also become a detriment by introducing the paradox of choice: when the number of choices increases, so does the difficulty of knowing what is best. I believe this introduces adoption hurdles when getting started with Orleans and configuring silos for production deployment and local development.

I know most, if not, all of this is covered by the docs, but it is spread out over multiple pages and can be overwhelming for a new user and can introduce friction for adoption. For example, the Project Setup page consists of multiple steps, all of which must be done properly in order to even get the project to compile and not crash at runtime

To remove friction, a set of dotnet new project templates would be created for common scenarios:

• Hosting a Silo alongside ASP.Net Core
• Hosting a Silo inside of a WorkerService
• Generating a classic 4-project solution (Console Client, Grain.Interfaces, Grains, Silo)
• Hosting a Silo (worker service) and a client (console) within Kuberenetes
• ... your suggestion here

The templates would take in some options, for example: --enable-transactions, --enable-azure-storage-tables-for-grain-storage, and --enable-mssql-for-grain-storage, --enable-tls, etc.

Recipes

It would be great to provide a library of use cases and how to accomplish them. We do have OrleansContrib/DesignPatterns, which are great, however the samples are stale and are abstracted at the level you'd expect in an architecture patterns book (I assume this was the point in of the repo).

The purpose of recipes would be to provide concrete step by step direction on how to accomplish certain tasks, explaining what they are doing, and why they are done this way, with links to the existing deeper dive docs.

A component of the recipies section would be a set of "I need X, use Y because of reasons", and provide guidance on how to accomplish it via recipes or documentation with explanations.

Some Examples:

• I need to receive events from a broker and send them to grains how should I do this?
• I need to call an external API that rate limits my calls, how do I do this?
• How can I call my grain via HTTP/gRPC?
• I saw Orleans doesn't have AuthN/AuthZ, how do I secure my silo and prevent rogue clients from connecting?
• Should I use Journaled Grains?
• How do I have an event sourced grain state?
• How do I modify multiple grains in a consistent way?
• When should I use Reentrant versus AlwaysInterleave?
• How can I enumerate my grains?
• ... your suggestion here

To supplement the recipe documentation, there would be corresponding dotnet new templates to generate the code needed for the use case.

Some example recipe templates I can think of off the top of my head:

• Work Queue: A grain with a persistent state that acts as a work queue using reminders
• Work Queue with Dispatcher: A grain with a persistent state that acts as a work queue using reminders to pass dequeued items to a target grain
• ... your suggestion here

My hope is that recipes will significantly decrease or remove the friction associated with Orleans adoption and remove some of the "gotchas" that lead to lost productivity.

I am looking for feedback in general and for scenarios/questions/template input from others that have spent time in the issue log and in the chat to help flesh out the topics.

I have identified some initial tasks for this work:

Recipes

• [ ] Introduce the Recipes section in docs with some basic use cases and answers gathered from chat
• [ ] .. TBD by discussion: choose a first recipe

Templates

• [ ] Create basic template that matches the Project Setup page
• [ ] .. TBD by discussion: choose other templates

[ElanHasson]

7175 for template example.

Related: https://github.com/dotnet/orleans/issues/7099#issuecomment-886651142

[ElanHasson]

Can linux and win cluster together? #7180 Good for recipe/q&a

[ElanHasson]

https://github.com/dotnet/orleans/issues/4694#issuecomment-890315467 for recipe/q&a "Both Akka and Akka.net seems to offer rebalancing when a new node is added to the cluster, does Orleans do the same or similar?"

[ElanHasson]

Best Practice For Modelling Aggregate Grains #7130

[ld100]

I'd say templates are fine, but having a solid documentation with non-primitive recipes is much more important.

I'm new to Orleans and so far trying to make it work is a constant struggle for me because of outdated and shallow docs. E.g. out of documentation you can set up classic 4-project solution just fine, but once I'm trying to do anything less common - I'm getting into trouble, within just 2 weeks I've got following issues:

1. Grain code was not picked by a client project till I did a hack with assembly reference, which was not mentioned in docs.
2. Unit tests does not work the way it is described in docs, at least when you're using ADO storage instead of default local one.
3. The silo does no start when co-hosted with the web host, despite being set up exactly as in docs. I'm still struggling with it.
4. Also had some issues setting up statefull grains till I found a solution looking up code examples. The problem was that the docs described only the easiest cases.

[ReubenBond]

@ld100 I agree with what you're saying. Could you open separate issues for documenting each of these and anything else which comes to mind? That's much easier to us to track and fix. Recipes are generally not whole applications, just task-oriented "how to" fragments.

@ElanHasson

4694 (comment) for recipe/q&a

"Both Akka and Akka.net seems to offer rebalancing when a new node is added to the cluster, does Orleans do the same or similar?"

I still don't understand what that commenter was actually referring to w.r.t Akka/.NET

[ElanHasson]

@ReubenBond

I still don't understand what that commenter was actually referring to w.r.t Akka/.NET

I agree on the ambiguity but for the purpose of Q&A parsed it (@serefarikan please correct me if I am wrong here) to be asking a few thing:

"Both Akka and Akka.net seems to offer rebalancing when a new node is added to the cluster,"

Understood as "Does Orleans rebalance active grains when a new silo is added?" "What if my active grains are too busy (high resource consumption) and are doing too much work in the silos they are currently activated on? Is there a way for them to be deactivated and activated on a less busy node or to rebalance the silo cluster's currently activated grains?"

I'm not very familiar with the details of Akka/Akka.Net, but i found https://petabridge.com/blog/cluster-sharding-technical-overview-akkadotnet/#:~:text=into%20a%20bottleneck.-,Rebalancing,-Rebalancing%20is%20a I think this functionality might be what the poster is referring to.

At the moment, I simply don't know if I can or cannot use Orleans for dynamic/elastic scaling of a cluster with persisted state. The documentation mentions Orleans can scale down when resources are not needed, but no mention of if you can scale up and no mention of how this interacts with backing storage either.

Understood as: "Orleans can scale down but not clear if it can scale up" (I believe they are referring to deactivations of grain when idle here).

It seems the idea of "scaling down" (grain deactivation) is being conflated with the ideas of "scaling up" (add a new silo) and "scaling out" (rebalancing the active grains on the newly added silo).

I am guessing that some folks get to Orleans via Akka/Akka.net and are happy to switch to a simpler programming model but are looking for some sort of "functional parity", at least for some of the more general things that may be important with distributed workloads (i.e.: rebalancing active work). I am speculating, but this guess is based on multiple people asking for similar things in that issue

My takeaways from #4694 in this issue context are answering the following:

Q: "Can an Orleans cluster be scaled at runtime?" A: "Yes, by adding new silos to the cluster."

Q: "Can Orleans be scaled down at runtime?" A: "Yes, by removing silos from the cluster."

Q: "When I add a new silo to the cluster, will active grains be moved in order redistribute the current loads to include the new load?" A: "No, Orleans does not yet support rebalancing active grain instances other nodes. However, any new grain activations may occur on the newly added silo and once the active grains from before expanding the cluster are deactivated, they may be reactivated on another silo when needed. You can achieve rebalancing by [...what can be done...]".

[serefarikan]

@ElanHasson apologies for such a late response. Same to @ReubenBond for leaving you in the dark by not clarifying my question after the helpful responses.

You nailed it with every item Elan. Yes, that's exactly what I meant when I asked my question. I'm also familiar with Akka/Akka.net and I was doing what you guessed: trying to figure out how/if some features are supported/not supported with Orleans.

Your answers would help people with profiles and intents similar to mine. Once again: thanks!

[ElanHasson]

7479 for templates