Support end-to-end testing for orchestrations involving durable entities

[brandonh-msft]

Is your feature request related to a problem? Please describe.

Currently time consuming to perform end-to-end testing for orchestrations involving durable entities.

Describe the solution you'd like Provide a mocking api that permits orchestration to invoke and engage durable entities from a testing framework like MSTest or XUnit.

The desired functionality would be the ability to create a unit test that exercises the scenarios outlined in the example. The test should engage not only the orchestration but also the dependent durable entities (instead of mocking the method invocations).

Describe alternatives you've considered

The only way that we can create such a test currently is to run the entire function app in a container with the functions runtime. Then, write tests that execute the functions from within the container. Alternatively (or additionally), the classical unit tests can only exercise the orchestration or entities individually.

Additional context

A code sample has been uploaded to the following github repo here.

This sample outlines a set of Acceptance Criteria that the orchestration and dependent durable entities should implement.

In order to test the end-to-end scenario in this bank account transfer scenario, the orchestration and durable entities must be run within the functions runtime connected to a storage account. This results in significant effort required to create tests that engage both the orchestration and dependent entities.

An example solution would be to provide a mocking api that, at minimum, simulates invoking the durable entities as the functions runtime does. Additionally, that mocking api should maintain durable entity state such that the entity's state from previous operations is maintained. Keeping that state In-memory would be acceptable for this use case. A storage account should not be required. This mocking api would include pre-built mocks for IDurableOrchestrationContext at minimum. Providing mocks for IDurableOrchestrationClient and IDurableEntityClient would further increase the code that could be engaged by the test such as HTTP Trigger that invokes durable entity and/or durable orchestrations

• If Durable Entities provided such a capability, this would be a distinguishing feature for the product. Such a feature is currently only provided by the Coyote project from MSR.
• Workarounds require significant effort both in their creation and maintenance as a result of needing to engage the functions runtime.

Sample Orchestration with acceptance test that this feature would permit to evaluate. Full sample here

public class TransfersHandler
{
    /* TODO: Write tests to exercise the following scenarios

        Scenario: Transfer
            Given account 1 with number 111 has balance of $100
            and account 2 with number 222 has balance of $50
            When transfer is made for $3 from account 1 to account 2
            Then account 1 should have a balance of $97
            and account 2 should have a balance of $53

        Scenario: Transfer more than available balance
            Given account 1 with number 111 has balance of $100
            and account 2 with number 222 has balance of $50
            When transfer is made for $200 from account 1 to account 2
            Then the transfer should be rejected due to overdraft

    */
    [FunctionName(nameof(ExecuteTransfer))]
    public async Task ExecuteTransfer([OrchestrationTrigger] IDurableOrchestrationContext context)
    {
        var message = context.GetInput<TransferMessage>();

        var fromAccountEntity = new EntityId(nameof(AccountEntity), message.FromAccountId);
        var toAccountEntity = new EntityId(nameof(AccountEntity), message.ToAccountId);

        using (await context.LockAsync(fromAccountEntity, toAccountEntity))
        {
            var fromAccountProxy = context.CreateEntityProxy<IAccountEntity>(fromAccountEntity);
            var toAccountProxy = context.CreateEntityProxy<IAccountEntity>(toAccountEntity);

            await fromAccountProxy.Debit(new DebitAccountMessage(message.Amount));
            await toAccountProxy.Credit(new CreditAccountMessage(message.Amount));
        }
    }
}

Internal Tracking

CSEF 359538

[sebastianburckhardt]

That's a very cool idea! I could see this develop in several steps.

• The first step would be to provide good hooks for mocking individual functions.
• The next step could be to publish a library that models executions faithfully (including serialization) but can run entirely in process without any I/O.
• The final step could be to make this work as a plug-in for Coyote. At that point we are way beyond unit testing but can test complex system behaviors using advanced techniques such as probabilistic concurrency testing. That would indeed be a distinguishing feature.