akkadotnet / akka.net

Canonical actor model implementation for .NET with local + distributed actors in C# and F#.
http://getakka.net
Other
4.72k stars 1.04k forks source link

No way to disable durable keys with DData when using Akka.Cluster.Sharding #7326

Open Aaronontheweb opened 3 months ago

Aaronontheweb commented 3 months ago

Version Information Version of Akka.NET? v1.5.27 and later Which Akka.NET Modules? Akka.Cluster.Hosting

Describe the bug

Here's the basic problem.

Both:

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder
            .WithDistributedData(options =>
            {
                options.Durable = new DurableOptions() { Keys = [] }; // disable persistence
            })
            .WithShardRegion<DrawingSessionActor>("drawing-session",
            (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
            new DrawingSessionActorMessageExtractor(), new ShardOptions()
            {
                StateStoreMode = StateStoreMode.DData,
                RememberEntities = true,
                RememberEntitiesStore = RememberEntitiesStore.Eventsourced,
                Role = clusterRoleName,
            });
        return builder;
    }
}

And

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder
            .WithDistributedData(options =>
            {
                options.Durable = new DurableOptions() { Keys = [] }; // disable persistence
            })
            .WithShardRegion<DrawingSessionActor>("drawing-session",
            (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
            new DrawingSessionActorMessageExtractor(), new ShardOptions()
            {
                StateStoreMode = StateStoreMode.DData,
                RememberEntities = true,
                RememberEntitiesStore = RememberEntitiesStore.Eventsourced,
                Role = clusterRoleName,
            });
        return builder;
    }
}

Still result in DData.Durable writing out durable keys and creating a new LMDB file.

Previously, the following command would have (edit: might have?) worked:

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder
            .WithShardRegion<DrawingSessionActor>("drawing-session",
            (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
            new DrawingSessionActorMessageExtractor(), new ShardOptions()
            {
                StateStoreMode = StateStoreMode.DData,
                RememberEntities = true,
                RememberEntitiesStore = RememberEntitiesStore.Eventsourced,
                Role = clusterRoleName, 
                DistributedData = { Durable = new DurableOptions() { Keys = [] }} // now-deprecated property
            });
        return builder;
    }
}

But that only worked because the WithShardRegion<T> command worked by applying HOCON globally, which was incorrect and wrong.

The suggested workaround for setting the ShardOptions.DistributedData property in the comments is to apply HOCON globally. That's what I'm effectively doing in the above code sample. However, this doesn't work because:

https://github.com/akkadotnet/akka.net/blob/355439ec5d2932385fcc23f1f6b14e1c2ff05ef1/src/contrib/cluster/Akka.Cluster.Sharding/ClusterShardingGuardian.cs#L302-L309

The sharding system will set durable keys anyway - so this makes me wonder, perhaps, if this is really an issue with Akka.Cluster.Sharding itself. I would love not to have an LMDB database created when I'm using remember-entities=on. I guess maybe we need to check the remember entities storage mode and not enable durable keys until we find one where that's explicitly set to ddata?

Expected behavior

I should be able to use remember-entities without LMDB instances being created unless I'm using DData for storage.

Actual behavior

I get an LMDB database no matter what I do.

Environment

Windows and Linux.

Aaronontheweb commented 3 months ago

cc @OnurGumus

Aaronontheweb commented 3 months ago

One design idea I had to fix this code is to do this:

internal static ReplicatorSettings GetReplicatorSettings(ClusterShardingSettings shardingSettings)
{
    var config = Context.System.Settings.Config.GetConfig("akka.cluster.sharding.distributed-data")
        .WithFallback(Context.System.Settings.Config.GetConfig("akka.cluster.distributed-data"));
    var configuredSettings = ReplicatorSettings.Create(config);
    var settingsWithRoles = configuredSettings.WithRole(shardingSettings.Role);
    if (shardingSettings.RememberEntities && shardingSettings.RememberEntitiesStore == RememberEntitiesStore.DData)
        return settingsWithRoles; // only enable durable keys when using DData for remember-entities
    else
        return settingsWithRoles.WithDurableKeys(ImmutableHashSet<string>.Empty);
}

That would stop the DData directory from being created unless we're explicitly using RememberEntitiesStore.DData as the remember-entities provider.

Another approached I've considered, and this is not mutually exclusive, is making it possible to pass in a ReplicatorSettings value as part of the ClusterShardingSettings class. This would make it possible to disable durable keys even when you're using DData as the remember-entities mode (which might be useful during local dev or testing.)

Aaronontheweb commented 3 months ago

Have a beta of this going into v1.5.28-beta1

Aaronontheweb commented 2 weeks ago

Can confirm that this is still not a totally solved problem in v1.5.30 - said as much when we merged #7327

Aaronontheweb commented 2 weeks ago

I can validate that this DOES disable DData from writing to disk:

public static class DrawingSessionActorExtensions
{
    public static AkkaConfigurationBuilder AddDrawingSessionActor(this AkkaConfigurationBuilder builder,
        string clusterRoleName = ClusterConstants.DrawStateRoleName)
    {
        builder.WithShardRegion<DrawingSessionActor>("drawing-session",
                (system, registry, resolver) => s => resolver.Props<DrawingSessionActor>(s),
                new DrawingSessionActorMessageExtractor(), new ShardOptions()
                {
                    StateStoreMode = StateStoreMode.DData,
                    RememberEntities = true,
                    RememberEntitiesStore = RememberEntitiesStore.DData,
                    Role = clusterRoleName
                })
            // .WithDistributedData(options =>
            // {
            //     options.Durable = new DurableOptions() { Keys = [] }; // disable persistence
            // })
            .AddHocon("akka.cluster.sharding.distributed-data.durable.keys = []", HoconAddMode.Prepend);
        return builder;
    }
}

It looks like this is an Akka.Hosting issue with options.Durable = new DurableOptions() { Keys = [] }; // disable persistence not working as expected - and that's probably an easy fix.