Upgrade to .NET 8 is breaking recovery

[wesselkranenborg]

Today we upgraded our service which is using Hyperion serialization to .NET 8. We are using Akka.Persistence.Azure and suddenly our state is broken.

We use Akka.NET 1.5.13 but I also tried with 1.5.14 and 1.5.15. The errors which we get (by only upgrading to .NET 8 with hyperion serialization) are:

Failed to deserialize instance of type . Failed to deserialize object of type [MyCompany.SystemStateService.Core.Actors.StaticStateActor+NewStateMessage] from the stream. Cause: Failed to deserialize object of type [System.Text.Json.JsonElement] from the stream. Cause: Failed to deserialize object of type [System.Text.Json.JsonDocument] from the stream. Cause: Unable to cast object of type 'System.Boolean' to type 'MetadataDb'. Failed to deserialize object of type [MyCompany.SystemStateService.Core.Actors.StaticStateActor+NewStateMessage] from the stream. Cause: Failed to deserialize object of type [System.Text.Json.JsonElement] from the stream. Cause: Failed to deserialize object of type [System.Text.Json.JsonDocument] from the stream. Cause: Unable to cast object of type 'System.Boolean' to type 'MetadataDb'. Failed to deserialize object of type [System.Text.Json.JsonElement] from the stream. Cause: Failed to deserialize object of type [System.Text.Json.JsonDocument] from the stream. Cause: Unable to cast object of type 'System.Boolean' to type 'MetadataDb'. Failed to deserialize object of type [System.Text.Json.JsonDocument] from the stream. Cause: Unable to cast object of type 'System.Boolean' to type 'MetadataDb'. Unable to cast object of type 'System.Boolean' to type 'MetadataDb'. 

System.Runtime.Serialization.SerializationException:
   at Akka.Serialization.HyperionSerializer.FromBinary (Akka.Serialization.Hyperion, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Serialization.Serialization.Deserialize (Akka, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Persistence.Serialization.PersistenceMessageSerializer.GetPayload (Akka.Persistence, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Persistence.Serialization.PersistenceMessageSerializer.GetPersistentRepresentation (Akka.Persistence, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Persistence.Serialization.PersistenceMessageSerializer.FromBinary (Akka.Persistence, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Serialization.Serializer.FromBinary (Akka, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Persistence.Azure.SerializationHelper.PersistentFromBytes (Akka.Persistence.Azure, Version=1.5.13.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Persistence.Azure.Journal.AzureTableStorageJournal+<ReplayMessagesAsync>d__21.MoveNext (Akka.Persistence.Azure, Version=1.5.13.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Akka.Persistence.Journal.AsyncWriteJournal+<>c__DisplayClass18_0+<<HandleReplayMessages>g__ExecuteHighestSequenceNr|0>d.MoveNext (Akka.Persistence, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.Runtime.Serialization.SerializationException handled at Akka.Serialization.HyperionSerializer.FromBinary:
   at Hyperion.ValueSerializers.ObjectSerializer.ReadValue (Hyperion, Version=0.12.2.0, Culture=neutral, PublicKeyToken=null)
   at Hyperion.Serializer.Deserialize (Hyperion, Version=0.12.2.0, Culture=neutral, PublicKeyToken=null)
   at Akka.Serialization.HyperionSerializer.FromBinary (Akka.Serialization.Hyperion, Version=1.5.14.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.Runtime.Serialization.SerializationException handled at Hyperion.ValueSerializers.ObjectSerializer.ReadValue:
   at Hyperion.ValueSerializers.ObjectSerializer.ReadValue (Hyperion, Version=0.12.2.0, Culture=neutral, PublicKeyToken=null)
   at lambda_method180 (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
   at Hyperion.ValueSerializers.ObjectSerializer.ReadValue (Hyperion, Version=0.12.2.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.Runtime.Serialization.SerializationException handled at Hyperion.ValueSerializers.ObjectSerializer.ReadValue:
   at Hyperion.ValueSerializers.ObjectSerializer.ReadValue (Hyperion, Version=0.12.2.0, Culture=neutral, PublicKeyToken=null)
   at lambda_method188 (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
   at Hyperion.ValueSerializers.ObjectSerializer.ReadValue (Hyperion, Version=0.12.2.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.InvalidCastException handled at Hyperion.ValueSerializers.ObjectSerializer.ReadValue:
   at lambda_method190 (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
   at Hyperion.ValueSerializers.ObjectSerializer.ReadValue (Hyperion, Version=0.12.2.0, Culture=neutral, PublicKeyToken=null)

This is what the StaticStateActor+NewStateMessage record does look like (it's not changed during the upgrade)

Is this a known thing? Are we forgetting something during upgrade?

I'm also curious why it is stating System.Text.Json in the exception as we are not using that as serializer. And also where type MetadataDb is coming from. That's nowhere mentioned in my code :-)

[wesselkranenborg]

@Aaronontheweb: is this something which is known? I'll try to create a minimal repro path this afternoon (CET). I'm the most curious about the System.Text.Json part, as we're using hyperion as serializer. This is our Akka.NET hosting code for serialization:

.WithCustomSerializer(
    "hyperion",
    new[] {
        typeof(object)
    },
    system => new HyperionSerializer(system))

[Aaronontheweb]

This is extremely odd, because we internally aren't using Systme.Text.Json anywhere - I wonder if the Azure Storage client itself is now returning something that is wrapped inside an STJ primitive.

[wesselkranenborg]

When I look into our state and base64 decode it, it shows this:

There you already see the System.Text.Json.* and also the MetadataDb (which is some internal type of System.Text.Json apparently).

We don't use System.Text.Json in the flow of this message but we use objects. Maybe somewhere internally these object properties are translated into System.Text.Json elements. But for now it doesn't look like something in Akka.Persistence.Azure, Hyperion or Akka.NET at all. Looks like a strange (de)serialization issue when using object properties.

So I suggest to close this issue.