search

willardf / Hazel-Networking

Hazel Networking is a low level networking library for C# providing connection-oriented, message-based communication via RUDP.
MIT License
391 stars 60 forks source link

Sent 6 pings that remote has not responded to #53

Closed aviv0099 closed 1 year ago

aviv0099 commented 1 year ago

Hello, I have the library in a game with over 3k daily active users and I get a lot of reports of disconnects. They get those errors:

More info:

Most of them are from Turkey, most of the players are able to play fine. The disconnects happen almost instantly to them(or after few seconds)

Any idea how can I fix it? Thank you!

willardf commented 1 year ago

Hmmm, regional disconnects is pretty tricky. We have already had to make small changes to the protocol to avoid some country-specific firewalls, so it's possible this could be another. But I haven't heard any significant reports from Among Us...

How many players are on the server at peak and how many packets are being sent? Is there any risk of CPU bottlenecking? Which ConnectionListener can also be a factor. Dtls is far more expensive than Udp.

aviv0099 commented 1 year ago

Hmmm, regional disconnects is pretty tricky. We have already had to make small changes to the protocol to avoid some country-specific firewalls, so it's possible this could be another. But I haven't heard any significant reports from Among Us...

How many players are on the server at peak and how many packets are being sent? Is there any risk of CPU bottlenecking? Which ConnectionListener can also be a factor. Dtls is far more expensive than Udp.

I use UdpConnectionListener, I also don't send a lot of packets - the tick rate is 10 packets per second for syncing the players. The problem happens when I have 30 ccu players(lowest) and I had no slowdowns when I had 700 ccu, its mainly for turkey players(some of them).(cpu is under 40% usage most of the time) I saw a lot of among us reports about this exact errors: https://www.reddit.com/r/AmongUs/comments/ioroxs/sent_6_pings_that_the_remote_has_not_responded_to/ https://www.reddit.com/r/AmongUs/comments/ihc9t8/sent_6_pings_that_remote_has_not_responded_to/ https://www.reddit.com/r/AmongUs/comments/j0033o/something_i_noticed_about_the_6_pings_issue/

Did you do something to fix it or it was fixed by itself? Can you point me to where I should start? Thank you for this amazing tool!

willardf commented 1 year ago

Yeah, so to be clear, these two errors are very common because they are the generic "Disconnected" error. Since you mentioned Turkey, I assumed that it might be regional since we have seen firewall issues in the past, but it doesn't really sound like that's the case here.

The other thing that could be regional-specific would be packet size. If you send a packet that is larger than about 800 bytes, you may start to see increased packet drop. This would get really bad up until 1200 bytes where it's nearly 100% drop rate. This is called MTU and Hazel has no built in features to fix it, but we did add a statistic to detect it: UdpConnection.Statistics.FragmentableMessagesSent. Looking at it now, you might want to adjust ConnectionStatistics.ExpectedMTU to be 800. As the code comments mention 576 is the smallest MTU, if you want to be extra strict, that's a good value to check too.

It definitely sounds like it's not a server issue. It's possibly a client error, but it would suggest that the game is hanging or failing to handle messages for multiple seconds, so that's unlikely. Therefore it's probably a network issue, which are very tricky to resolve. Could also be a bug, but that also will take a lot of digging to figure out.

Really hopeful that MTU tracking helps because I'm afraid I won't be too much help otherwise.

aviv0099 commented 1 year ago

Yeah, so to be clear, these two errors are very common because they are the generic "Disconnected" error. Since you mentioned Turkey, I assumed that it might be regional since we have seen firewall issues in the past, but it doesn't really sound like that's the case here.

The other thing that could be regional-specific would be packet size. If you send a packet that is larger than about 800 bytes, you may start to see increased packet drop. This would get really bad up until 1200 bytes where it's nearly 100% drop rate. This is called MTU and Hazel has no built in features to fix it, but we did add a statistic to detect it: UdpConnection.Statistics.FragmentableMessagesSent. Looking at it now, you might want to adjust ConnectionStatistics.ExpectedMTU to be 800. As the code comments mention 576 is the smallest MTU, if you want to be extra strict, that's a good value to check too.

It definitely sounds like it's not a server issue. It's possibly a client error, but it would suggest that the game is hanging or failing to handle messages for multiple seconds, so that's unlikely. Therefore it's probably a network issue, which are very tricky to resolve. Could also be a bug, but that also will take a lot of digging to figure out.

Really hopeful that MTU tracking helps because I'm afraid I won't be too much help otherwise.

Thanks a lot, and thanks for the explanation! I will put into the client this statistic + more like packet loss and that will help me identify the problem. btw this is the game, was a pleasure using this library! https://store.steampowered.com/app/1904820