Using development connection fails when attempting to check for Azure storage

[go-aegian]

Getting this error when the connection is development and nothing is setup to run on azure

akka.persistence.snapshot-store.tester#1153123657]: Akka.Actor.ActorInitializationException: Exception during creation ---> System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) ---> System.NullReferenceException: Object reference not set to an instance of an object at Akka.Persistence.Azure.Snapshot.AzureBlobSnapshotStore.InitCloudStorage(Int32 remainingTries)

It might be this line needs to be using null interpolation in case nothing is setup

            if (credentialSetup**?**.HasValue)

[Arkatufus]

Can you post your (sanitized) HOCON configuration? and are you using the new setup class?

[go-aegian]

Basically the hocon remained the same when it's local so in this case is when it fails, just upgraded the package and run it locally and it failed.

tester {

qualified type name of the Azure Blob Storage persistence snapshot storage actor

    class = "Akka.Persistence.Azure.Snapshot.AzureBlobSnapshotStore, Akka.Persistence.Azure"

    # connection string, as described here: https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string
    connection-string = "UseDevelopmentStorage=true"

    # the name of the Windows Azure Blob Store container used to persist snapshots
    container-name = "default"

    #  Initial timeout to use when connecting to Azure Storage for the first time.
    connect-timeout = 3s

    # Timeouts for individual read, write, and delete requests to Azure Storage.
    request-timeout = 3s

    # Toggle for verbose logging. Logs individual requests to Azure.
    # Intended only for debugging purposes. akka.loglevel must be set to DEBUG
    # in order for this setting to be effective.
    verbose-logging = off

    # dispatcher used to drive snapshot storage actor
    plugin-dispatcher = "akka.actor.default-dispatcher"
  }

[Arkatufus]

Can you rename tester to akka.persistence.snapshot-store.azure-blob-store and see if it works?

[Arkatufus]

You also need to set akka.persistence.snapshot-store.plugin = "akka.persistence.snapshot-store.azure-blob-store" in your configuration.

[Arkatufus]

I may have found the problem, we will try to do a patch/small release and see if it fixes your problem.

[go-aegian]

great let me know I'll update the package and test it

[Aaronontheweb]

https://github.com/petabridge/Akka.Persistence.Azure/releases/tag/0.8.1 - now live on NuGet @sainzg

[Arkatufus]

@sainzg please re-open this issue if the problem persists

[go-aegian]

I have another class that deals straights to blob storage and after securing it, is able to read/write blobs

but when it comes to akka, I think I'm making a mistake somewhere as it seems the actor fails to connect to the container when restoring its state from snapshotoffer messages, or for some reason it does not use the container-name specified in hocon?

---

app.conf - partial

snapshot-store { plugin = "akka.persistence.snapshot-store.azure-blob-store"

  actor-blob {
    # qualified type name of the Azure Blob Storage persistence snapshot storage actor
    class = "Akka.Persistence.Azure.Snapshot.AzureBlobSnapshotStore, Akka.Persistence.Azure"

    # connection string, as described here: https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string
    connection-string = ""

    # the name of the Windows Azure Blob Store container used to persist snapshots
    container-name = "actorblobcontainer"

    #  Initial timeout to use when connecting to Azure Storage for the first time.
    connect-timeout = 3s

    # Timeouts for individual read, write, and delete requests to Azure Storage.
    request-timeout = 3s

    # Toggle for verbose logging. Logs individual requests to Azure.
    # Intended only for debugging purposes. akka.loglevel must be set to DEBUG
    # in order for this setting to be effective.
    verbose-logging = off

    # dispatcher used to drive snapshot storage actor
    plugin-dispatcher = "akka.actor.default-dispatcher"
  }

}

---

Main.cs - hoconConfig will be loaded/boostrapped from app.conf above

public static void Main(string[] args) {

actorSystem = ActorSystem.Create(ClusterHelper.ActorSystem, hoconConfig);
actorSystem.Settings.Setup.And(AzureBlobSnapshotSetup.Create(new Uri(blobConnection), new DefaultAzureCredential()));
var initialActor = actorSystem.ActorOf(InitialActor.Props(), nameof(InitialActor));

}

---

initial actor - partial

public class InitialActor : ReceivePersistentActor {

public InitialActor() {
    SnapshotPluginId = "akka.persistence.snapshot-store.actor-blob";
        Recover<SnapshotOffer>(HandleEvent);
}

private void HandleEvent(SnapshotOffer evt) {
      if (evt.Snapshot is somestateclass) ....
}

}

[go-aegian]

Just to give you more insight, even when the blob storage is not locked down to private mode, and I basically don't even set this line actorSystem.Settings.Setup.And(AzureBlobSnapshotSetup.Create(new Uri(blobConnection), new DefaultAzureCredential())); The current package fails to restore snapshots so for the time being I'm rolling it back to 0.7.2 which works.

Hope this help and please let me know when you have another release and I'll test it out again and provide you with feedback

[Arkatufus]

Can you change the way your ActorSystem is initialized? I think its supposed to be:

var bootstrap = BootstrapSetup.Create()
    .WithConfig(hoconConfig)
    .And(AzureBlobSnapshotSetup.Create(new Uri(blobConnection), new DefaultAzureCredential()));

actorSystem = ActorSystem.Create(ClusterHelper.ActorSystem, bootstrap);
var initialActor = actorSystem.ActorOf(InitialActor.Props(), nameof(InitialActor));

[go-aegian]

@Arkatufus

Ok I did set it up that way you recommended and kept the version as 0.8.1.

Here are my findings

  - Local storage works, as it uses the local connection and does not add the AzureBlobSnapshotSetup class to it.
        Created new actor and saved its state, then updated state all fine.

  - Cloud storage does not work. 
        Configured to be open as always with Blobs set as Public access level set to Container and Blob Storage open to public read.

        This is for a brand new actor which does not have anything in the journal and/or snapshot blobs.

        Error:    Persistence failure when replaying events for persistenceId [Test-e12556ad-81cb-4a3c-9339-c988588c64c5]. Last known sequence number [0]2021-04-22T17:25:09.330528449ZAkka.Persistence.RecoveryTimedOutException: Recovery timed out, didn't get snapshot within 30s.

[Arkatufus]

Are there any other Azure related errors before this happens, does the persistence plugin manage to connect correctly to the service? Have you set all of the credentials correctly?

[Arkatufus]

Have you set up at least one of the proper way of connecting to Azure service using DefaultAzureCredential? The list of types of credential setup can be read here

[go-aegian]

Yes on the Azure side all is setup correctly because I ran another service outside Akka that reads/writes and uploads new blobs to another container, I can either secure the entire storage or just the container, connect using the same DefaultAzureCredential and all works just fine.

[Arkatufus]

@sainzg I've looked further into this and found no problem with our code. Here is the code I've used to test that everything works: Code

[go-aegian]

Thank you so much. I will be able to test it again in a few weeks.