Message-associated Metadata Values are now of type JsonElement after v5 update

[Laiteux]

Hey! It's me again (:

Just stumbled upon a bug in the library, which was introduced with v5. I assume this is related to the migration from Newtonsoft.Json to System.Text.Json.

Despite Messages/SyncRequests Metadata being of type Dictionary<string, object>, these objects actually seem to be JsonElements.

While previously we could simply cast them to the desired type straight away (for example using (string)), we are now forced to cast them to JsonElement first, then to do something like .GetString() if we are expecting a string:

string commandName = ((JsonElement)message.Metadata["cmd"]).GetString()!;

I doubt this is intended (since it didn't work that way before the migration), and if it is then it surely would be clearer to have them as JsonElement straight away, instead of object.

[jchristn]

Hi @Laiteux acknowledged and on my list to test, sorry it will be a little rough time-wise this week due to the holiday.

[Laiteux]

Obviously no problem, enjoy your holiday! And happy thanksgiving (:

[Laiteux]

I'd like to add that Metadata is never null anymore after v5 update.

Being able to check if it was null was very useful. For example, this is how I was using it to my advantage back with v4:

public static TResponse HandleResponse<TResponse>(SyncResponse response)
{
    if (response.Metadata != null) // is exception
    {
        var exceptionType = (string)response.Metadata[ExceptionTypeMetadataKeyName];
        var exceptionMessage = (string)response.Metadata[ExceptionMessageMetadataKeyName];

        throw new TcpCallbackCommandException(exceptionType, exceptionMessage);
    }

    return TcpCommand.DeserializeData<TResponse>(response.Data);
}

Since v5, even if no metadata is provided, it will still be not null.

Is this intended? If so, why? Currently, this instead forces me to check if a key exists in the Metadata Dictionary, which is way less convenient.

[jchristn]

Hi @Laiteux can you help me produce the metadata always being not null issue? I'm not seeing that.

C:\Code\Watson\WatsonTcp\src\Test.Server\bin\Debug\net7.0>test.server
Server IP: [localhost]
Server port: [9000]
Use SSL: [y/N]?
[Info     ] [WatsonTcpServer] starting on 127.0.0.1:9000
Server started
Command [? for help]: [Debug    ] [WatsonTcpServer] accepted connection from [3f6ebe9b-e9ee-4f20-b5d0-ba94611e6482|127.0.0.1:58852]
[Debug    ] [WatsonTcpServer] starting data receiver for [3f6ebe9b-e9ee-4f20-b5d0-ba94611e6482|127.0.0.1:58852]
Client connected: [3f6ebe9b-e9ee-4f20-b5d0-ba94611e6482|127.0.0.1:58852]
Message from [3f6ebe9b-e9ee-4f20-b5d0-ba94611e6482|127.0.0.1:58852]: Hello from the client
Metadata: (null)

I amended the MessageReceived event handler in both Test.Server and Test.Client to include:

            if (args.Metadata == null)
            {
                Console.WriteLine("Metadata: (null)");
            }
            else
            {
                Console.Write("Metadata: ");
                if (args.Metadata.Count < 1)
                {
                    Console.WriteLine("(none)");
                }
                else
                {
                    Console.WriteLine(args.Metadata.Count);
                    foreach (KeyValuePair<string, object> curr in args.Metadata)
                    {
                        Console.WriteLine("  " + curr.Key.ToString() + ": " + curr.Value.ToString());
                    }
                }
            }

[jchristn]

Also I'm able to directly .ToString() the metadata:

Client sending the message:

C:\Code\Watson\WatsonTcp\src\Test.Client\bin\Debug\net7.0>test.client
Server IP: [localhost]
Server port: [9000]
Use SSL: [y/N]?
[Info     ] [WatsonTcpClient] connecting to localhost:9000
Server connected
[Info     ] [WatsonTcpClient] connected to localhost:9000
Command [? for help]: send md
Data: Message from the client
Key  : Key1
Value: Value1
Key  : Key2
Value: Value2
Key  :
Command [? for help]:

Server receiving the message:

C:\Code\Watson\WatsonTcp\src\Test.Server\bin\Debug\net7.0>test.server
Server IP: [localhost]
Server port: [9000]
Use SSL: [y/N]?
[Info     ] [WatsonTcpServer] starting on 127.0.0.1:9000
Server started
Command [? for help]: [Debug    ] [WatsonTcpServer] accepted connection from [de599526-3a23-49af-b479-26c1518c1aff|127.0.0.1:58885]
[Debug    ] [WatsonTcpServer] starting data receiver for [de599526-3a23-49af-b479-26c1518c1aff|127.0.0.1:58885]
Client connected: [de599526-3a23-49af-b479-26c1518c1aff|127.0.0.1:58885]
Message from [de599526-3a23-49af-b479-26c1518c1aff|127.0.0.1:58885]: Message from the client
Metadata: 2
  Key1: Value1
  Key2: Value2

Server code handling the metadata (similar on client):

if (args.Metadata == null)
            {
                Console.WriteLine("Metadata: (null)");
            }
            else
            {
                Console.Write("Metadata: ");
                if (args.Metadata.Count < 1)
                {
                    Console.WriteLine("(none)");
                }
                else
                {
                    Console.WriteLine(args.Metadata.Count);
                    foreach (KeyValuePair<string, object> curr in args.Metadata)
                    {
                        Console.WriteLine("  " + curr.Key.ToString() + ": " + curr.Value.ToString());
                    }
                }
            }

[Laiteux]

Uh.. You're right. I'm not able to reproduce this anymore. Really not sure what happened and how. I was still a little high too so that probably didn't help.

Very sorry about this. I'll let you know if I somehow stumble upon it again.

[jchristn]

Code does strange things whilst high :) I'll close for now! PS I hope it was a good strain.

[Laiteux]

Hahah!

You just closed this as completed btw, what about the original issue?

[jchristn]

I might need help reproducing this.

In the MessageReceived event handler:

if (args.Metadata.ContainsKey("foo")) Console.WriteLine(args.Metadata["foo"]);

I'm not having to go through JsonElement.

Client:

Server:

[Laiteux]

That is because Console.WriteLine is using the ToString() implementation of JsonElement.

This will output True: Console.WriteLine(args.Metadata["foo"] is JsonElement);

Trying to store it as a variable using a cast however won't work: (string)args.Metadata["foo"]

Instead, you are required to do this: string foo = ((JsonElement)args.Metadata["foo"]).GetString()!;

[jchristn]

I see what you're saying now.

On the client, I added:

                    case "send md":
                        userInput = Inputty.GetString("Data:", null, false);
                        metadata = Inputty.GetDictionary<string, object>("Key  :", "Value:");
                        metadata.Add("time", DateTime.UtcNow);
                        if (!_Client.Send(Encoding.UTF8.GetBytes(userInput), metadata)) Console.WriteLine("Failed");
                        break;

And on the server:

                if (args.Metadata.ContainsKey("time"))
                {
                    DateTime timestamp = (DateTime)(args.Metadata["time"]);
                    Console.WriteLine(timestamp.ToString());
                }

Which results in:

C:\Code\Watson\WatsonTcp\src\Test.Server\bin\Debug\net7.0>test.server
Server IP: [localhost]
Server port: [9000]
Use SSL: [y/N]?
[Info     ] [WatsonTcpServer] starting on 127.0.0.1:9000
Server started
Command [? for help]: [Debug    ] [WatsonTcpServer] accepted connection from [a8ec9fb7-1fa6-4fa9-ad4d-a125ffcb94b9|127.0.0.1:59947]
[Debug    ] [WatsonTcpServer] starting data receiver for [a8ec9fb7-1fa6-4fa9-ad4d-a125ffcb94b9|127.0.0.1:59947]
Client connected: [a8ec9fb7-1fa6-4fa9-ad4d-a125ffcb94b9|127.0.0.1:59947]
Message from [a8ec9fb7-1fa6-4fa9-ad4d-a125ffcb94b9|127.0.0.1:59947]: message
Metadata: 2
  hello: world
  time: 2022-11-28T23:05:03.8860444Z
[Error    ] Event handler exception in MessageReceived:
System.InvalidCastException: Unable to cast object of type 'System.Text.Json.JsonElement' to type 'System.DateTime'.
   at TestServer.TestServer.MessageReceived(Object sender, MessageReceivedEventArgs args) in C:\Code\Watson\WatsonTcp\src\Test.Server\Server.cs:line 301
   at WatsonTcp.WatsonTcpServerEvents.<>c__DisplayClass40_0.<HandleMessageReceived>b__0() in C:\Code\Watson\WatsonTcp\src\WatsonTcp\WatsonTcpServerEvents.cs:line 141
   at WatsonTcp.WatsonTcpServerEvents.WrappedEventHandler(Action action, String handler, Object sender) in C:\Code\Watson\WatsonTcp\src\WatsonTcp\WatsonTcpServerEvents.cs:line 176

I'm not sure there is a fix that can be applied here, however, you can always use Newtonsoft.Json by implementing your own ISerializationHelper and assigning that instance to WatsonTcpServer.SerializationHelper and WatsonTcpClient.SerializationHelper.

One of the main motivations for moving away from Newtonsoft.Json was/is dependency drag (and conflicts) - the current implementation should allow you to revert back to v4.x behavior though.

[Laiteux]

I'm entirely in favor of System.Text.Json, that's not a problem 😉

However, considering all metadata values will now be JsonElements (I believe?), don't you think we should move away from object and use JsonElement straight away as the metadata dictionary value type? (Dictionary<string, JsonElement>)

That would make everything clearer, avoiding misunderstandings or even breaking-without-warning changes when upgrading from v4 to v5. Because at least, it wouldn't let you compile anymore with simple casts after upgrading:

[jchristn]

I don't disagree - at all - but my worry is that 1) it may be too inflexible and 2) it prevents someone being able to attach metadata to messages when they don't use the built-in System.Text.Json serializer/deserializer. Thoughts?

[Laiteux]

Well, I think you're right, and I just noticed why: Because you are now (since v5) allowing for different serializers to be implemented (using ISerializationHelper), there is no other way around using object on both the sending and receiving side.

I guess this issue can therefore safely be closed?

[Laiteux]

On a side note, I think it would be great if it would be possible to access a default JsonSerializerOptions instance from DefaultSerializationHelper.

This would allow the user to modify it to its liking, or even use it himself to serialize stuff (what I do) in the exact same manner as the library (with the exact same options, naming policy, indentation etc...).

I checked and you're not even using a custom JsonSerializerOptions instance, so the default one I would add would simply be = JsonSerializerOptions.Default, therefore not being a weird or even breaking change, if that makes sense.

What do you think about this? And would you like me to get a PR done for it?

[jchristn]

Hi @Laiteux I just published v5.0.7 for this if you'd like to give it a try and let me know!

NuGet: https://www.nuget.org/packages/WatsonTcp/5.0.7 Commit: https://github.com/jchristn/WatsonTcp/commit/28d2655b3a53922860524fa4597020e903eb3ab0

[Laiteux]

Thanks!

However:

(I haven't changed anything in my code, and reverting back to v5.0.6 eliminates the issue)

[jchristn]

Ouch. Ok, this is an issue with exposing bool pretty = true in the signature. I can only (apparently) set the WriteIndented property one time.

This JsonSerializerOptions instance is read-only or has already been used in serialization or deserialization.

The workaround would be to include JsonSerializerOptions in the method signature, but then that would be specific to System.Text.Json.

BTW, I've pulled v5.0.7. I'm inclined to leave that as an implementation detail to the developer (managing their own JsonSerializerOptions and just reverting to creating a new instance (with WriteIndented set/unset) each time SerializeJson is called.

Thoughts?

[Laiteux]

Sounds good to me.

Well, I guess we're (finally) done then!

Thanks a ton for your work on all of this, Joel. It was a pleasure (:

[jchristn]

Likewise Matt, much appreciated! Cheers