Unit Testing Orleans Applications

[akashlal]

Hi All, I'm a researcher in MSR. I'm fairly new to Orleans, but I have been working on "design and testing of asynchronous applications" for some time. I wanted to ask a question and then request for feedback on some of our recent work if you think it is relevant for Orleans developers.

Question: How do people do "unit testing" of Orleans applications? Are there standard mocks (for grains, etc.) that allow testing without production deployment? I've noticed this question come up several times in discussion forums for Reliable Actors (in Service Fabric).

Feedback Requested: We have done some work ourselves: https://github.com/p-org/PSharpModels We have mocks of Orleans grains (as well as ReliableActors of Fabric). These allow testing of (largely) unmodified applications without deployment. We also have tools that do robust testing by covering many different interleavings and interactions between different grains, even covering non-determinism from timers. (Think CHESS or Cuzz, if you are familiar with those tools from MSR.) The underlying technology is P#: https://github.com/p-org/PSharp

Our modeling of Orleans is far from complete, but we have sorted out many of the basic issues and have started pushing real code through. Would anyone be interested in using this? We'd love to engage with early adopters. Is this even relevant or are there better solutions out there?

Thanks, we would appreciate any feedback! Akash & the P# team

[sergeybykov]

@akashlal So glad to see you here! :-)

There are two main approaches for unit testing Orleans apps.

1) Using test by hosting one or more silos in separate app domains. Look at how TestCluster class is used in the Tester project. This approach uses 100% of the Orleans runtime, but within a single process.

2) Grain classes can be instantiated directly by passing them mocks of IGrainIdentity and IGrainRuntime. Here Orleans runtime is completely out of the picture.

[shayhatsor]

@akashlal, I've read a bit about P# and if I understand correctly, it is what Orleans unit testing has been missing. The first approach @sergeybykov has mentioned does use 100% of the Orleans runtime, but it doesn't reflect an Orleans production deployment. Most importantly, it has a designation of a primary silo, which doesn't exist in production where all are secondary silos. The primary silo is a single point of failure which holds the in-memory runtime system stores (membership and reminders). This is where P# comes to the rescue by removing the difference between a silo that runs in development or production mode. Leveraging the model, we'll be able to simulate duplicate activations, silo crashes, memory pressure, grain placements, network delays, etc. All of which are priceless!

[akashlal]

Thanks for your comments @sergeybykov and @shayhatsor.

We would like to contribute a serious test infrastructure to Orleans. Reading around a bit, I see others calling this integration testing rather than unit testing, but model exploration will be a new feature.

Ideally we should go through a rigorous exercise before we claim to understand the entire problem. Are there Orleans applications that you would like tested? Are there parts of Orleans that are implemented as Orleans services that we can dig into?

Calling out to @philbe and @sebastianburckhardt with whom I've discussed this in person as well.

Thanks, Akash

[sergeybykov]

@shayhatsor

The first approach @sergeybykov has mentioned does use 100% of the Orleans runtime, but it doesn't reflect an Orleans production deployment. Most importantly, it has a designation of a primary silo, which doesn't exist in production where all are secondary silos.

This should be easy to update to use the peer to peer model, by configuring tests silos to use a real membership provider, e.g. AzuReTable-based. Or by adding a tests one.

@akashlal

Reading around a bit, I see others calling this integration testing rather than unit testing, but model exploration will be a new feature.

Yeah, we had these discussions early on on unit vs. integration testing. It doesn't really matter for us how one calls it. :-)

Are there Orleans applications that you would like tested?

We have a number of internal users. The immediate question from them I can predict will be "what do we need to do for that beside giving you access to our sources?"

Are there parts of Orleans that are implemented as Orleans services that we can dig into?

Not clear to me what "Orleans services" means exactly in this context. We do have a number on Orleans Runtime components implemented as internal partitioned services leveraging system targets (system grains) as distribution and concurrency primitives.

[shayhatsor]

@sergeybykov

This should be easy to update to use the peer to peer model, by configuring tests silos to use a real membership provider, e.g. AzuReTable-based. Or by adding a tests one.

I know, but I considered all non in-memory membership providers as a production setup. @akashlal asked:

How do people do "unit testing" of Orleans applications? Are there standard mocks (for grains, etc.) that allow testing without production deployment?

Per my understanding, he was interested in test procedures that do not require external dependencies, i.e. tests that exercise the code in a similar fashion to P#. Where the code runs against a model, which virtualizes everything in-memory.

[akashlal]

@sergeybykov

The immediate question from them I can predict will be "what do we need to do for that beside giving you access to our sources?"

We'll need enough information to set up test cases. This means at least: sources, instructions to build, a pointer to the Orleans-based component, and existing test cases for it (i.e., assertions or specifications they'd like tested). Even better if they already do unit-testing in the way you suggested.

It is usually difficult to set up tests for others. But we'd be willing to try.

@shayhatsor

code runs against a model, which virtualizes everything in-memory.

That's exactly right. This is necessary to make things fast and the scheduling controllable.

[dsarfati]

I have already added some additional capability (1.2.2) for unit testing without running a real Orleans runtime. It is simply an evolution on the optional grain constructor that allows you to inject a mocked/stubbed runtime into the grain. Our team is in the process of ramping up our testing using the framework I added.

We decided to break tests into 3 categories: Unit- Stubbed/mocked Orleans runtime. Used to test that each grain is behaving as expected (correct number of writes, correct data sent over streams, calls to other (mocked) grains. Behavior- Run a full Orleans cluster (real storage provider, multiple silos...) with a real asp.net web api. The test driver then makes calls as if it was a client application and asserts the correct values are returned. End to End - Full Orleans cluster, real asp.net web api a,d real UI. Test driver clicks buttons and asserts entire system is acting as expected.

[sergeybykov]

@akashlal We can take this offline. I can connect you to a couple of groups near us.

[meenanarendra]

Do we have sample that we can refer to implement unit testing?

[sergeybykov]

@meenanarendra Are you asking about unit testing a grain class outside of the silo execution environment or within it?

[meenanarendra]

@sergeybykov We are developing a application and want to know the a good approach to unit test the code. I want to know your suggestion for both approach. You can also provide me sample code to achieve the same.

[RehanSaeed]

It would be really nice if Orleans had something akin to the TestHost API in ASP.NET Core which allows you to bootstrap the app, make calls to it in-memory and replace services with mocked ones where necessary by override ConfigureServices.

[dsarfati]

I believe this is possible with the TestCluster. We are looking at providing these types of mocked services as part of the TestKit

[RehanSaeed]

Yes, I discovered that late on Friday and promptly put it to use in my new Orleans dotnet new project template at https://github.com/Dotnet-Boxed/Templates/blob/master/Docs/Orleans.md.

[fjallemark]

The examples of using TestCluster seems to be invalid for Orleans version 2. There are no parameterless constructor for TestCluster, see https://dotnet.github.io/orleans/Documentation/tutorials_and_samples/testing.html. The exampe should be changed to for example Cluster = new TestClusterBuilder().Build();

[veikkoeeva]

There's one example by @JorgeCandeias at https://jorgecandeias.github.io/ too. I recently opened one PR https://github.com/dotnet/orleans/pull/5909 in which a cluster in reliable configuration is started. The difference to in production should be change in connection strings and something like using managed identities. That's just a PR still, not necessarily accepted, and especially the docs isn't ready. Shows one way of doing it, though.