Proto Cluster with Consul shows connection errors

[danfma]

For local development, I've configured Proto Cluster with Consul support using random ports.

The cluster is working and I have two nodes:

• An API Server which acts as a bridge (WebSocket to Actor cluster bridge);
• A Lobby room server which is responsible for the game;

All nodes are communicating with each other normally, but the Lobby server is showing some RPC exceptions like if it wasn't able to communicate with the API server, but both are green on Consul as you can see in the second image.

[rogeralsing]

This looks suspicious. Why does it say localhost and not 127.0.0.1 which is what is registered in Consul? Can you show the remote config for both nodes? are there any advertised host set?

[rogeralsing]

Could it be that you are manually constructing a PID that points to localhost and try to communicate with that?

[danfma]

This looks suspicious. Why does it say localhost and not 127.0.0.1 which is what is registered in Consul? Can you show the remote config for both nodes? are there any advertised host set?

So, for local development, which is this case, I'm using the GrpcNetRemoteConfig.BindToLocalhost()

And consul is being configured using the default settings:

Note, this also happens with the K8 provider but usually there it becomes more stable with time:

[rogeralsing]

Is there any chance you are sending messages that are not supported by the serializers? I don´t quite see how there can be RpcExceptions for nodes that are otherwise healthy.

Are the logs above the only logs that appear? nothing else?

[danfma]

Yes, I'm using custom serializers but, in theory, I should have received a warning if an unknown message was attempt to be serialized. I've put a fallback serializer to raise a warning if I'm serializing an unknown message type, and I can't see any warnings in the logs...

public sealed class AnyMemoryPackableSerializer(ILogger logger) : ISerializer
{
    public ByteString Serialize(object obj)
    {
        var bytes = MemoryPackSerializer.Serialize(obj.GetType(), obj);

        logger.LogWarning(
            "Serializing {Type} by AnyMemoryPackableSerializer resulting in {Bytes} bytes",
            obj.GetType().Name,
            bytes.Length
        );

        return bytes.ToByteStringDangerously();
    }

    public object Deserialize(ByteString bytes, string typeName)
    {
        logger.LogWarning(
            "Deserializing {Type} by AnyMemoryPackableSerializer loading {Bytes} bytes",
            typeName,
            bytes.Length
        );

        var type = Type.GetType(typeName)!;

        return MemoryPackSerializer.Deserialize(type, bytes.Span)
            ?? throw new InvalidOperationException($"Failed to deserialize message of type {typeName}");
    }

    public string GetTypeName(object message)
    {
        return message.GetType().AssemblyQualifiedName!;
    }

    public bool CanSerialize(object obj)
    {
        return obj is IMemoryPackFormatterRegister;
    }
}

public sealed class NonPackableErrorSerializer(ILogger logger) : ISerializer
{
    public ByteString Serialize(object obj)
    {
        logger.LogCritical("Attempt to serialize {Type} by NonPackableErrorSerializer", obj.GetMessageTypeName());

        throw new NotSupportedException();
    }

    public object Deserialize(ByteString bytes, string typeName)
    {
        logger.LogCritical("Attempt to deserializing {Type} by NonPackableErrorSerializer", typeName);

        throw new NotSupportedException();
    }

    public string GetTypeName(object message)
    {
        return message.GetType().AssemblyQualifiedName!;
    }

    public bool CanSerialize(object obj)
    {
        return true;
    }
}

   private static GrpcNetRemoteConfig AddSerializers(GrpcNetRemoteConfig remoteConfig, ILoggerFactory loggerFactory)
    {
        var serializerId = 2;
        var priority = 100;

        return remoteConfig
            .WithProtoMessages()
            .WithSerializer(
                serializerId: serializerId++,
                priority: priority--,
                serializer: new MemoryPackableSerializer<GameEvent>(typeName: nameof(GameEvent))
            )
            .WithSerializer(
                serializerId: serializerId++,
                priority: priority--,
                serializer: new MemoryPackableSerializer<GameCommand>(typeName: nameof(GameCommand))
            )
            .WithSerializer(
                serializerId: serializerId++,
                priority: priority--,
                serializer: new MemoryPackableSerializer<ILobbyRoomActivatorMessage>(
                    typeName: nameof(ILobbyRoomActivatorMessage)
                )
            )
            .WithSerializer(
                serializerId: serializerId++,
                priority: priority--,
                serializer: new MemoryPackableSerializer<ILobbyRoomMessage>(typeName: nameof(ILobbyRoomMessage))
            )
            .WithSerializer(
                serializerId: serializerId++,
                priority: priority--,
                serializer: new MemoryPackableSerializer<ILobbyParticipantMessage>(
                    typeName: nameof(ILobbyParticipantMessage)
                )
            )
            .WithSerializer(
                serializerId: serializerId++,
                priority: priority--,
                serializer: new MemoryPackableSerializer<RemotePid>(typeName: nameof(RemotePid))
            )
            .WithSerializer(
                serializerId: serializerId++,
                priority: priority,
                serializer: new AnyMemoryPackableSerializer(
                    logger: loggerFactory.CreateLogger<AnyMemoryPackableSerializer>()
                )
            )
            .WithSerializer(
                serializerId: serializerId,
                priority: -1, // NOTE -1 means after all other serializers but before the JsonSerializer
                serializer: new NonPackableErrorSerializer(
                    logger: loggerFactory.CreateLogger<NonPackableErrorSerializer>()
                )
            );
    }