Error "Cannot find serializer with id [210] (manifest [WTRACE])" when remoting from an F# script

[object]

I am trying to communicate from an F# script with an actor that runs in an Akka.NET cluster. The script references a few NuGet packages, i.e.

#r "nuget: Akka"
#r "nuget: Akka.Remote"
#r "nuget: Akka.Serialization.Hyperion"
...

The configuration of an Akka system in the script mostly matches the cluster:

let config =
    """
    akka { 
        actor {
            provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
            serializers {
                hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
            }
            serialization-bindings {
                "System.Object" = hyperion
            }
            serialization-identifiers { 
                "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion" = -5
            }
        }
        remote {
          dot-netty.tcp {
            hostname = ""
            tcp-reuse-addr = on
          }
        }
    """

I succesfully connect to an actor in a cluster by remote actor path ("akka.tcp://%s@%s:%d/user/globalconnectpriorityqueue/singleton") attempt to exchange messages with it fails, I see following errors in the log:

[WARNING][04/17/2024 12:00:49.954Z][Thread 0043][akka.tcp://Test@0.0.0.0:1999/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FOddjob%40maodatest01.felles.ds.nrk.no%3A1972-1/endpointWriter/endpointReader-akka.tcp%3A%2F%2FOddjob%40maodatest01.felles.ds.nrk.no%3A1972-1] Deserialization failed for message with serializer id [210] and manifest [WTRACE]. Transient association error (association remains live). Cannot find serializer with id [210] (manifest [WTRACE]). The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems. Serializer Id [210] is not one of the internal Akka.NET serializer. Please contact your plugin provider for more information.. Serializer Id [210] is not one of the internal Akka.NET serializer. Please contact your plugin provider for more information. Cause: System.Runtime.Serialization.SerializationException: Cannot find serializer with id [210] (manifest [WTRACE]). The most probable reason is that the configuration entry 'akka.actor.serializers' is not in sync between the two systems. Serializer Id [210] is not one of the internal Akka.NET serializer. Please contact your plugin provider for more information. at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest) at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, Payload message, IActorRef senderOption) at Akka.Remote.EndpointReader.b__11_0(InboundPayload inbound)

I've found an issue related to serializer 201 (https://github.com/petabridge/phobos-issues/issues/57). I suspected in my case this can be due to missing DLLs, so I referenced all Akka and Phobos DLLs plus DLLs used by remoting (like Dotnetty). Still same error.

Any idea of what can cause this error?

[Arkatufus]

Can you change your configuration to this and see if it fixes your issue?

let config = ConfigurationFactory.Parse(
    """
    akka { 
        actor {
            provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
            serializers {
                hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
            }
            serialization-bindings {
                "System.Object" = hyperion
            }
            serialization-identifiers { 
                "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion" = -5
            }
        }
        remote {
            dot-netty.tcp {
                hostname = ""
                tcp-reuse-addr = on
            }
        }
    }
    """).WithFallback(ActorTracing.DefaultConfig())

or you can embed the Phobos.Tracing configuration directly:

let config =
    """
    akka 
    {
        actor 
        {
            provider = "Akka.Remote.RemoteActorRefProvider, Akka.Remote"
            serializers 
            {
                hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
                phobos-tracing = "Phobos.Tracing.Serialization.TraceEnvelopeSerializer, Phobos.Tracing"
            }
            serialization-bindings 
            {
                "System.Object" = hyperion
                "Phobos.Tracing.IWithTrace, Phobos.Tracing" = phobos-tracing
            }
            serialization-identifiers 
            {
                "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion" = -5
                "Phobos.Tracing.Serialization.TraceEnvelopeSerializer, Phobos.Tracing" = 210
            }
        }
        remote 
        {
            dot-netty.tcp 
            {
                hostname = ""
                tcp-reuse-addr = on
            }
        }
    }
    phobos.tracing
    {
        serialization
        {
            # Defines the OpenTracing format we're going to use for serializing
            # and deserializing ISpan instances over the wire. 
            # Valid values for this setting are `text_map` and `http_headers`.
            format = text_map

            # Defines the OpenTracing carrier used for serialization and deserialization of spans.
            # Valid values for this setting are `text_map`.
            carrier = text_map
        }

        # Specifies whether or not we're going to use the ActorScopeManager or not
        use-actorscopemanager = on
    }
    """

[Aaronontheweb]

@object I think your script needs to reference Phobos also in order to make sure it has the serializer definitions present - the responses you're getting back from your Phobos-enabled cluster are what's causing the problem because they include trace data.

[object]

@Arkatufus @Aaronontheweb Thank you for a quick response. Yes, this did the trick. Providing a fallback for Phobos configuration helped. But we don't use WithFallback(ActorTracing.DefaultConfig()) in our code, yet it all works. Is this because we set up the ActorSystem using PhobosConfigBuilder so we resolve tracing config implicitly? Should we as a general rule always include fallback to ActorTracing.DefaultConfig when we have Phobos tracing enabled?

[Aaronontheweb]

. Is this because we set up the ActorSystem using PhobosConfigBuilder so we resolve tracing config implicitly?

That's correct - that will always implicitly load the tracing serializer.

[object]

Makes sense, thanks!