Performance issues under poor network conditions

ghost commented 2 years ago

Godot version

3.3.4.stable

Plugin version

0.5

System information

Windows 10, Linux Mint 20.2

Issue description

Under poor network conditions (high latency and packet loss), WebRTCMultiplayer causes frame rate issues in the Godot engine. The issue seems to manifest itself on nodes that are receiving packets. Using NetworkedMultiplayerENet under these conditions does not have the issue. I was able to reproduce the issue on both Windows and Linux.

https://user-images.githubusercontent.com/93407490/139504250-5e1e1fc7-a258-49a6-87d0-f92ab8581353.mp4

Steps to reproduce

Establish a connection between two peers using WebRTCMultiplayer.
Simulate latency and packet loss
- I used Clumsy on Windows to set latency to 400ms and packet loss to 25%, but I also had the issue occurring at 250ms latency and 2% packet loss
- I was also able to reproduce on Linux using tc

Minimal reproduction project

WebRTC Performance Issue.zip

ghost commented 2 years ago

After some more testing, I was also able to reproduce the issue by exporting the project to HTML5 (I should have tried this sooner, apologies), so I'm not sure if the issue is in the Godot engine proper or this plugin.

I also tried adjusting the Max Channel In Buffer Kb project setting, but could not notice a difference with a larger buffer.

dsnopek commented 2 years ago

I haven't ever seen this manifest as a "frame rate" or "performance" problem, but I can say that WebRTC is hyper sensitive to packet loss. In my testing, if you introduce even a little bit of packet loss (like 2%) you'll see messages get really delayed, way more than you'd expect, and certainly more than compared to ENet (which I've also done comparison testing with).

I believe is caused by the flow control that's built into the SCTP protocol (which is what WebRTC uses for data channels). I think it's detecting the packet loss, and then holding back messages, probably because the protocol is assuming the packet loss is caused by network congestion. And if you try to send more messages after it's entered "flow control mode," it'll actually exacerbate the problem, and cause it to hold back the message even longer. I've observed this with both HTML5 and with the native plugin.

Perhaps this just looks like a frame rate or performance problem because your game objects are moving in fits and starts rather than smoothly?

I've had some success by checking WebRTCDataChannel.get_buffered_amount() and having my code avoid sending if there's any data buffered, which appears to get back out of "flow control mode" quicker. However, that function was only just added in this PR:

https://github.com/godotengine/webrtc-native/pull/43

... so, there isn't a pre-compiled binary that you can download. (Also, this appears to work more effectively with the native plugin than it does in HTML5, at least in my testing.)

Hopefully @Faless will make a new release of the native plugin shortly after Godot 3.4 stable is released. :-)

dsnopek commented 2 years ago

The description on this PR has some decent references about WebRTC, its flow control and bufferedAmount: https://github.com/godotengine/godot/pull/50658

ghost commented 2 years ago

Thanks for the insight. Your comment made me curious if the built-in browser RTCPeerConnection behaved the same way under the poor conditions, and lo and behold it does. The issue definitely seems to be the SCTP protocol itself rather than Godot, so thanks for the details surrounding that. I will keep my eye on get_buffered_amount and use it once it's generally available.

For reference, I've included the projects I was testing with. One is the Godot code exported to HTML5, and the other is a pure JavaScript implementation of the same thing. RtcPerformance.zip

godotengine / webrtc-native