Unit testing the Actors

[mandjeo]

I wrote an application that uses dapr Actors in dotNet core. I'm trying to unit test the actors and I currently see no way to do it. I used ActorHost.CreateForTest method to create the host and set the ID, however when my Actor gets the state it fails with the InvalidOperationException and the following message:

The actor was initialized without a state provider, and so cannot interact with state. If this is inside a unit test, replace Actor.StateProvider with a mock.

As far as I see StateProvider property is available on the ActorHost but it's internal and can't be set from the outside.

Also, in my actor I use the Reminders feature, and I would like to unit test that part as well.

Am I missing something and what would be the intended way to unit test the code that uses the actors? Also, if it's not possible to mock the necessary components in order to write a unit test, how would you suggest to setup the infrastructure so that I can do integration tests with dapr runtime and the actors while also being able to debug my tests?

Any help is appreciated!

[edmondbaloku]

Did you find any solution to this @mandjeo ?

[mandjeo]

@edmondbaloku

So, what we ended up doing is the following:

• create a mock of the ActorTimerManger which we pass to the ActorTestOptions in order to be able to test things related to reminders
• use the ActorHost.CreateForTest method to create the host pass ActorTestOptions with timer manager mock as well as the ActorId
• introduce an internal constructor which receives IActorStateManager and then sets the protected StateManager property
• in the tests we use this internal constructor to pass the stub implementation of the state manager (we implemented the stub with an in memory state management)

These enabled us to do basic unit testing of the functionality with in the actor. However, it gets very tricky when you are relying on actor reminders to be executed during the test or when you need to work with protected fields/methods.

Even though I really like the Actors concept from dapr, I think a bit more attention should be put on making it more testable.

Also, I would love to hear from someone from the dapr team on how are we supposed to run integration tests which include this functionality, to give a (even high level) idea on how should we setup our test infrastructure for that?

[m3nax]

Below is a brief example of how I managed to write some unit tests. It use XUnit and Moq. I hope it can help you.

/// <summary>
/// Persistent state for the storage actor.
/// </summary>
public class StorageState
{
    /// <summary>
    /// List of items in the storage.
    /// </summary>
    public Collection<string> Items { get; set; } = [];
}

public class StorageActor : Actor, IStorage
{
    /// <summary>
    /// The name of the state used to store the storage state.
    /// </summary>
    public const string StateName = "state";

    /// <summary>
    /// Initializes a new instance of <see cref="Storage"/>.
    /// </summary>
    /// <param name="host"></param>
    /// <param name="actorStateManager">Used in unit test.</param>
    public StorageActor (ActorHost host, IActorStateManager? actorStateManager = null)
        : base(host)
    {
        if (actorStateManager != null)
        {
            this.StateManager = actorStateManager;
        }
    }

    /// <inheritdoc/>
    public async Task Add(ICollection<string> items)
    {
        ArgumentNullException.ThrowIfNull(items, nameof(items));

        if (items.Count == 0)
        {
            throw new InvalidOperationException("Sequence contains no elements");
        }

        var state = await this.StateManager.GetStateAsync<StorageState>(StateName);

        // Add items to the storage.
        foreach (var item in items)
        {
            state.Items.Add(item);
        }

        await this.StateManager.SetStateAsync(StateName, state);
    }
}

public class StorageTests
{
    [Fact]
    public async Task Add_EmptyItemsCollection_ThrowInvalidOperationException()
    {
        // arrange
        var mockStateManager = new Mock<IActorStateManager>(MockBehavior.Strict);
        var host = ActorHost.CreateForTest<StorageActor>();

        var itemsToAdd = new List<string>();
        var storageActor = new StorageActor(host, mockStateManager.Object);

        // act
        var act = () => storageActor.Add(itemsToAdd);

        // assert
        var ex = await Assert.ThrowsAsync<InvalidOperationException>(act);
        Assert.Equal("Sequence contains no elements", ex.Message);
    }

        [Fact]
    public async Task Add_AddItemsToStorage()
    {
        // arrange
        var mockStateManager = new Mock<IActorStateManager>(MockBehavior.Strict);
        var host = ActorHost.CreateForTest<StorageActor>();

        var itemsToAdd = new List<string> { "item1", "item2" };
        var storageActor = new StorageActor(host, mockStateManager.Object);
        var storageState = new StorageState
        {
            Items = new Collection<string>
            {
                "item0",
            }
        };

        mockStateManager
            .Setup(x => x.GetStateAsync<StorageState>(It.IsAny<string>(), It.IsAny<CancellationToken>()))
            .ReturnsAsync(storageState);

        mockStateManager
            .Setup(x => x.SetStateAsync(It.IsAny<string>(), It.IsAny<StorageState>(), It.IsAny<CancellationToken>()))
            .Returns(Task.CompletedTask);

        // act
        await storageActor.Add(itemsToAdd);

        // assert
        mockStateManager.Verify(x => x.SetStateAsync(
            Storage.StateName,
            It.Is<StorageState>(x => x.Items.Count == 3 && x.Items.Intersect(itemsToAdd).Count() == itemsToAdd.Count),
            It.IsAny<CancellationToken>()),
            Times.Once);
    }
}

[m3nax]

In the next few days I will write an example project

[m3nax]

/assign

[paule96]

@m3nax is this really the correct way to do that? Wouldn't it be better to extend the method CreateForTest by a parameter for the Actor host's internal StateProvider property? So it would be possible, without changing the implementation, to test it?

Note: That will be maybe also complicated because there is no interface for DaprStateProvider and that is an internal type

[paule96]

@m3nax for your pull request maybe another idea. I know the following solutions uses reflection what is not nice. But it avoids changing the implementation of the real actor.

var actor = new StartEventActor(host, daprClient);
actor.GetType()
    .GetProperty(nameof(StartEventActor.StateManager))!
    .SetValue(actor, new TestActorStateManager());

It uses reflection because the setter of StateManager is not accessible

And then you need an implementation for IActorStateManager.

using Dapr.Actors.Runtime;
using System.Collections.Concurrent;

namespace Hmp.nGen.ProcessEngine.ActorHost.Tests.TestHelpers
{
    internal class TestActorStateManager : IActorStateManager
    {
        private ConcurrentDictionary<string, string> stateStore = new();
        public Task<T> AddOrUpdateStateAsync<T>(string stateName, T addValue, Func<string, T, T> updateValueFactory, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(addValue);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task<T> AddOrUpdateStateAsync<T>(string stateName, T addValue, Func<string, T, T> updateValueFactory, TimeSpan ttl, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(addValue);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task AddStateAsync<T>(string stateName, T value, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task AddStateAsync<T>(string stateName, T value, TimeSpan ttl, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task ClearCacheAsync(CancellationToken cancellationToken = default)
        {
            stateStore = new();
            return Task.CompletedTask;
        }

        public Task<bool> ContainsStateAsync(string stateName, CancellationToken cancellationToken = default)
        {
            return Task.FromResult(stateStore.ContainsKey(stateName));
        }

        public Task<T> GetOrAddStateAsync<T>(string stateName, T value, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.GetOrAdd(stateName, valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task<T> GetOrAddStateAsync<T>(string stateName, T value, TimeSpan ttl, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.GetOrAdd(stateName, valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task<T> GetStateAsync<T>(string stateName, CancellationToken cancellationToken = default)
        {
            if (stateStore.TryGetValue(stateName, out var resultString))
            {
                return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
            }
            throw new KeyNotFoundException($"The key '{stateName}' was not found in the statestore");
        }

        public Task RemoveStateAsync(string stateName, CancellationToken cancellationToken = default)
        {
            stateStore.Remove(stateName, out _);
            return Task.CompletedTask;
        }

        public Task SaveStateAsync(CancellationToken cancellationToken = default)
        {
            return Task.CompletedTask;
        }

        public Task SetStateAsync<T>(string stateName, T value, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task SetStateAsync<T>(string stateName, T value, TimeSpan ttl, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(System.Text.Json.JsonSerializer.Deserialize<T>(resultString)!);
        }

        public Task<bool> TryAddStateAsync<T>(string stateName, T value, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(true);
        }

        public Task<bool> TryAddStateAsync<T>(string stateName, T value, TimeSpan ttl, CancellationToken cancellationToken = default)
        {
            var valueString = System.Text.Json.JsonSerializer.Serialize(value);
            var resultString = stateStore.AddOrUpdate(stateName, valueString, (_, _) => valueString);
            return Task.FromResult(true);
        }

        public Task<ConditionalValue<T>> TryGetStateAsync<T>(string stateName, CancellationToken cancellationToken = default)
        {
            throw new NotImplementedException();
        }

        public Task<bool> TryRemoveStateAsync(string stateName, CancellationToken cancellationToken = default)
        {
            throw new NotImplementedException();
        }
    }
}

That is really just a stupid test implementation that is only useful if you want to test a single actor.

[paule96]

@m3nax if you are already working on it, did you find a good way to mock CreateProxy calls on the ActorHost inside an actor? Where the ActorHost was create with ActorHost.CreateForTest(...)

Because my actors do often things like:

var actor = this.ProxyFactory.CreateActorProxy<IMyActor>(ActorId.CreateRandom(), nameof(MyActor));
await actor.MyMethod();

I mean you can specify a mock of the ActorProxy. But I don't know if this is the correct abstraction to use. Maybe there is something similar like ActorHost.CreateForTest(...) for the ActorProxy too.

In my most basic tests, I just use: ActorProxy.DefaultProxyFactory.

But with this method, all calls to actors just end with errors. (ofc, it can't find the implementation of the target actor without additional configuration)

[m3nax]

@m3nax is this really the correct way to do that? Wouldn't it be better to extend the method CreateForTest by a parameter for the Actor host's internal StateProvider property? So it would be possible, without changing the implementation, to test it?

Note: That will be maybe also complicated because there is no interface for DaprStateProvider and that is an internal type

A more native approach that doesn't require a change in the contract is the best solution, so far as I understand, what I've implemented is the only way I've found to test

@m3nax for your pull request maybe another idea. I know the following solutions uses reflection what is not nice. But it avoids changing the implementation of the real actor.

Right now I'm on vacation when I get back I'll try to find a more solid solution to propose. Perhaps with reflection, perhaps by adding an overload to the CreateForTest method.

@m3nax if you are already working on it, did you find a good way to mock CreateProxy calls on the ActorHost inside an actor? Where the ActorHost was create with ActorHost.CreateForTest(...)

Because my actors do often things like:

var actor = this.ProxyFactory.CreateActorProxy<IMyActor>(ActorId.CreateRandom(), nameof(MyActor));
await actor.MyMethod();

I mean you can specify a mock of the ActorProxy. But I don't know if this is the correct abstraction to use. Maybe there is something similar like ActorHost.CreateForTest(...) for the ActorProxy too.

In my most basic tests, I just use: ActorProxy.DefaultProxyFactory.

But with this method, all calls to actors just end with errors. (ofc, it can't find the implementation of the target actor without additional configuration)

As for the case of a call to one actor within a method of another I have to think/find out how to do it.

@philliphoff do you have any idea?

[philliphoff]

do you have any idea?

@m3nax

I haven't done a lot of unit testing with actors (calling actors), so I don't have an immediate answer. I wonder if one could pass the actor an Func<Actor, IActorProxyFactory> (or equivalent) proxy interface. The actor would use this proxy interface rather than use its own proxy factory directly. The default implementation (for production) would simply return the actor's own proxy factory (e.g. actor => actor.ProxyFactory), effectively keeping the same behavior. But, when testing, you could pass an implementation that returned a mocked proxy factory.

It's a bit annoying to have to introduce an additional layer, I agree. It seems like, ideally, the ActorHost created "for test" should offer a settable proxy factory, or perhaps AppHost in an actor should be an interface rather than a concrete type (which is probably a more invasive and breaking change).

[paule96]

I now do something similar to what @philliphoff recommends todo. So my setup for the ProxyFactory looks now like that:

var actorMoq = new Mock<IMyActor>();
var proxyFactoryMoq = new Moq.Mock<IActorProxyFactory>();
proxyFactoryMoq.Setup(p => p.CreateActorProxy<IMyActor>(
    It.IsAny<ActorId>(),
    It.IsAny<string>(),
    It.IsAny<ActorProxyOptions>())
)
.Returns(actorMoq.Object);

var host = ActorHost.CreateForTest(typeof(ActorToTest), new()
{
    ProxyFactory = proxyFactoryMoq.Object
});

if you want to return special mocks for some special actor type you can do that:

proxyFactoryMoq.Setup(p => p.CreateActorProxy<IProcessActor>(
    It.IsAny<ActorId>(),
    It.Is<string>(s => s == nameof(MyActor)),
    It.IsAny<ActorProxyOptions>())
)
.Returns(actorMoq.Object);

With this you can define multiple mocks in the proxy factory with different behaviors