Some messages are missing

[renato-kristic]

I use this library to receive messages from specific clients and then broadcast them to all connected clients. Lately I noticed that some messages get lost sometimes so I created three scripts that help me to stress test this library:

• One script is sending counter message every second so I can have track of messages that are sent
• Second script sends larger messages (about 2KB) every 100ms

• Third script is just client that is monitoring messages that are exchanging

  After some time (after ~1000 messages are exchanged) I notice that some messages are missing. Counter script successfully send messages but receiving script sometimes does not receive few messages.

  When I check debug messages I can see all messages that get exchanged. I can see event=write message that are emitted multiple times. (Probably for each client connected) At the moment that some messages are missing I can see that these messages are not emitted the same number of times. Number of emitted messages are random, sometimes message is sent to 10 clients, sometimes to only 3.

  When I stop script that is sending messages every 100ms and wait for few minutes I can see that SignalR server recovers and counter messages start arriving in correct order.

To eliminate issues with client I tried running .Net version of SingalR server and I can confirm that with this server everything is working just fine.

Any ideas what is causing these issues?

[philippseith]

can you please create a sample project?

[renato-kristic]

I did extensive testing to narrow down root cause of this problem. Multiple clients were used to receive messages. (python, rust, and golang) One of the golang clients was using different signalR library. (experimental library, not this one) This client was using delay in receiving callback method and it was causing all other clients to have delay in receiving messages and also dropping some messages. As soon as I terminate this experimental golang client everything becomes responsive again. I'll just replace this defective golang client with this one and it will solve my problems.

But strange thing is that other clients should not brake server functionality. As mentioned in post above, when I try signalr server implemented in .net (Microsoft.AspNet.SignalR) and run the same clients I cannot reproduce this problem.

Your client will solve my problem but it will be great if you can check if there is something implemented wrong with the server as it can start behaving strange when different client is used. (experimental one that most likely has bug in it)

[philippseith]

can you share a link to this experimental golang client? And can you share some sample code how you were calling back the client?

[renato-kristic]

Sorry, this library is not open-source so I cannot share it here.

This is an example how server is broadcasting messages to all clients.

func(c * SignalRTest) Messaging(message json.RawMessage) { var receivedMessage map[string] interface {} errUnmarshal: = json.Unmarshal([] byte(message), & receivedMessage) if errUnmarshal != nil { fmt.Println(errUnmarshal) return } c.Clients().All().Send("messaging", receivedMessage) }

If creator of this experimental client library figures out what is casing this problem I'll share it here.

[renato-kristic]

I analyzed this defective signalr client library that I used and found out that signalR loop can be blocked by message handler that our project consumes. When I modified this library to execute message handler method in go routine all problem disappeared. (loop is not blocked anymore)

Blocking main loop probably caused blocking of websocket connection. Maybe server is waiting for some kind of ACK from client and client is not responding instantly. (just my guess).

In my test, after 1000 messages exchanged (2kb size), memory usage of server started climbing rapidly. After few hours server consumed 1.5GB of memory.

When .Net signalr server is used I cannot reproduce these problems. Can you check what can causing your server to behave like this if websocket connection of client's library is blocked for some reason?

I solved my problem by using your signalr client, but it will be great if this server can also be fixed and to make it more robust. Defective client should not break server...

[philippseith]

The different behavior of the ASP.NET server and this one might be caused by this method: https://github.com/philippseith/signalr/blob/21ec27f84141c894238d30ae60e5de9e85f0e2ec/hublifetimemanager.go#L48

It loops through all clients and sends blocking. If the one client blocks, all the following are blocked also.

[renato-kristic]

Is it possible to implement it in non-blocking way? It will be great if this library can behave just as .Net version.

[philippseith]

yes, it is. I will look into it in the next days

[philippseith]

@renato-kristic please check if https://github.com/philippseith/signalr/tree/bugfix/%23170_Non_blocking_Invoke solves the problem for you

[renato-kristic]

@philippseith thanks for you effort. I did the same test using experimental signalr golang client with a test script that delays execution of receiver callback method. I can confirm that this fixes problem with delays in receiving messages by other clients and no messages are lost. :)

But I also noticed that memory usage of server is also climbing rapidly. (just as before) After exchanging few thousands of messages server start consuming few hundreds of MB. This only happens if I use defective client that blocks signalr loop. Also, this client has delay in receiving messages as only one message arrives in between signalr loop delay.

When I stop sending messages memory usage starts dropping slowly.

[philippseith]

no wonder, the client invocation is made asynchronous with one goroutine per invoke. Each goroutine takes about 4k memory