search

akkadotnet / akka.net

Canonical actor model implementation for .NET with local + distributed actors in C# and F#.
http://getakka.net
Other
4.69k stars 1.04k forks source link

serialize-messages =on doesn't work with DistributedPubSub #4120

Open MaximG1234 opened 4 years ago

MaximG1234 commented 4 years ago

Akka.NET 1.4.0-beta3

It appears that enabling 'serialize-messages = on' while using clustering breaks the actor system with the following error.

Looks like clustering has some difficulty correctly serializing its messages when turned on.

Here is a project which reproduces the error https://www.dropbox.com/s/o9al6adwl609p7o/AkkaTest.zip?dl=0


[ERROR][22/12/2019 12:32:41 AM][Thread 0005][[akka://PubSubSystem/system/cluster/core#342317816]] Swallowing exception during message send
Cause: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type System.Collections.Immutable.ImmutableHashSet`1[System.Type]. Path 'To.$values', line 1, position 376.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewList(JsonReader reader, JsonArrayContract contract, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Akka.Serialization.NewtonSoftJsonSerializer.FromBinary(Byte[] bytes, Type type)
   at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)
[DEBUG][22/12/2019 12:32:41 AM][Thread 0005][akka://PubSubSystem/system/clusterEventBusListener] Started (Akka.Cluster.ClusterReadView+EventBusListener)
[ERROR][22/12/2019 12:32:41 AM][Thread 0019][[akka://PubSubSystem/system/cluster/core#342317816]] Swallowing exception during message send
Cause: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type System.Collections.Immutable.ImmutableHashSet`1[System.Type]. Path 'To.$values', line 1, position 368.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewList(JsonReader reader, JsonArrayContract contract, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Akka.Serialization.NewtonSoftJsonSerializer.FromBinary(Byte[] bytes, Type type)
   at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)
[DEBUG][22/12/2019 12:32:41 AM][Thread 0019][akka://PubSubSystem/system/remote-watcher] Started (Akka.Cluster.ClusterRemoteWatcher)
[DEBUG][22/12/2019 12:32:41 AM][Thread 0009][EventStream] subscribing [akka://PubSubSystem/system/cluster/core/daemon#73339279] to channel Akka.Remote.QuarantinedEvent
[ERROR][22/12/2019 12:32:41 AM][Thread 0007][[akka://PubSubSystem/system/cluster/core#342317816]] Swallowing exception during message send
Cause: Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type System.Collections.Immutable.ImmutableHashSet`1[System.Type]. Path 'To.$values', line 1, position 379.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewList(JsonReader reader, JsonArrayContract contract, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Akka.Serialization.NewtonSoftJsonSerializer.FromBinary(Byte[] bytes, Type type)
   at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
   at Akka.Actor.ActorCell.SerializeAndDeserializePayload(Object obj)
   at Akka.Actor.ActorCell.SerializeAndDeserialize(Envelope envelope)
   at Akka.Actor.ActorCell.SendMessage(Envelope message)
MaximG1234 commented 4 years ago

Can confirm that the cluster system does not correctly form and end up getting the following at the publisher. If I turn serialization back off, everything works as expected.

[INFO][22/12/2019 12:44:39 AM][Thread 0005][Cluster (akka://PubSubSystem)] Cluster Node [akka.tcp://PubSubSystem@192.168.0.29:123] - Received InitJoin message from [[akka.tcp://PubSubSystem@0.0.0.0:3212/system/cluster/core/daemon/joinSeedNodeProcess-1#1491684597]], but this node is not initialized yet [INFO][22/12/2019 12:44:44 AM][Thread 0005][Cluster (akka://PubSubSystem)] Cluster Node [akka.tcp://PubSubSystem@192.168.0.29:123] - Received InitJoin message from [[akka.tcp://PubSubSystem@0.0.0.0:3212/system/cluster/core/daemon/joinSeedNodeProcess-1#1491684597]], but this node is not initialized yet

Aaronontheweb commented 4 years ago

@MaximG1234 thanks for reporting this - thought we had fixed it already in https://github.com/akkadotnet/akka.net/pull/3725 but that must not be the case.

Aaronontheweb commented 4 years ago

https://github.com/akkadotnet/akka.net/pull/3791 might contain a fix

Aaronontheweb commented 4 years ago

So this isn't an issue with Akka.Remote or Akka.Cluster - it appears to be an issue with Akka.Tools.DistributedPubSub, based on your reproduction sample. I'm going to push a reproduction spec here shortly.

Aaronontheweb commented 4 years ago

Taking this off of the 1.4.0 milestone - it's a minor issue and ultimately a problem because Newtonsoft.Json can't handle Systems.Collections.Immutable. Not getting a clear beat on which specific messages are causing the issue in #4721 but I'll leave this up for grabs.