Godot RC4 Linux Server causing high latency between clients over ws connections.

[notmushroomoverhere]

Godot version: v3.2.beta6 and RC2 OS/device including version: Windows 10 Pro

Issue description: Severe latency issues when using the 3.2beta6 build of Godot to create a websocket connection between Linux hosted servers and client previews, over ws.

I exported a Linux server pck file and HTML5 export from the stable v3.1.2 Godot build. Ran two clients on 2 webpages (as well as on 2 Godot engine previews) and the server on a Linux Virtual Machine, using the Godot_v3.1.2-stable_linux_server.64. I successfully connected the clients to the server, and saw no noticeable latency.

Using the exact same project, I exported a second Linux server pck file, but using the 3.2beta6 Godot build. I then ran the new Linux server on the same Virtual Machine, but used the 3.2beta6 Linux Server (more info below) and ran the clients inside 3.2beta6 Godot engine previews. I achieved a successful connection, but there was very noticeable latency between the 2 clients, leading me to believe that the beta version of the Godot Linux server is causing high latency across websocket connections.

I've tested across multiple machines, different IP's and Virtual Machines, and know of someone else who is experiencing the same issue.

Steps to reproduce:

• Setup a Linux Virtual Machine - can be done for free on Google Cloud Platform. I followed this tutorial: https://gitlab.com/menip/godot-multiplayer-tutorials/blob/master/GCPTutorial/GCPTut.md
• Download the min_server project and extract.
• Download the min_client project and extract.
• OPTIONAL: Download the server_pck_files, which should save you time.

Using the Godot v3.2.beta6:

• Open the min_client project, and in gamestate.gd, write the port and IP on line 4 and 5 that is linked to your Linux virtual machine. (Ensure that the connection is ws://IP:port)
• Export the min_server project as a .pck, or use the one that I prepared in server_pck_files, labelled 'beta_server'
• Run the server project on the Linux Virtual Machine, using the Linux 64 server found here: https://downloads.tuxfamily.org/godotengine/3.2/beta6/
• Run 2 client instances in 2 separate Godot editors.
• Have both connect to the Linux server by pressing the 'Join' button.
• Once both have joined, press the 'Ready' button on each client.
• Run the two clients side by side, and you should notice some very obvious latency.

OPTIONAL (In case you want to see the version without latency.)

• The same server and client projects are used as above, but a different .pck is used.
• This time, export the server pck from the stable Godot v3.1.2, or use the one that I prepared, labelled 'stable_server'.
• Run the server project on the Linux Virtual Machine, using the standard Linux server from the Godot website: https://godotengine.org/download/server, as opposed to the beta build one.
• Run 2 client instances in 2 Godot editors.
• Have both connect to the Linux server by pressing the 'Join' button.
• Once both have joined, press the 'Ready' button on each client.
• Run the two clients side by side, and you should notice no latency.

Minimal reproduction project:

min-client.zip min-server.zip server_pck_files.zip

[Calinou]

This sounds like a duplicate of #23898 to me.

[notmushroomoverhere]

This sounds like a duplicate of #23898 to me.

I'm not sure that it is, as I did have ws working with no latency on 3.1. Perhaps I am misunderstanding how development builds work, but it would seem to me that on the other post, an older build was used compared to the beta build that I am using. I would've thought it would be an issue with this current development build (I heard there was some kind of switch to mbedtls - could that perhaps be why?)

[Calinou]

(I heard there was some kind of switch to mbedtls - could that perhaps be why?)

The switch to mbedTLS was completed before 3.1 was released :slightly_smiling_face:

[notmushroomoverhere]

(I heard there was some kind of switch to mbedtls - could that perhaps be why?)

The switch to mbedTLS was completed before 3.1 was released 🙂

Ah my mistake.

[notmushroomoverhere]

Just wanted to update after reading 'In particular, any change that would cause a regression in your projects is very important to report (e.g. if something that worked fine in 3.1.x no longer works in 3.2) from the Godot RC2 newspost.' - I tested on RC2 server export and the RC2 Linux server, and the high latency still remains. On 3.1, I did not experience any latency when connecting over ws.

EDIT: De-activating Vsync as suggested in #23898 visibly reduced latency, but it is still higher latency than in 3.1.

EDIT 2: Turning Vsync off in the project settings reuslted in a relative low latency when the game was previewed in the editor. However, exporting this project to HTML5 and hosting it on a webpage resulted in very high latency again.

[Faless]

@notmushroomoverhere can you try out #35516 ?

[notmushroomoverhere]

@Faless Perhaps I am misunderstanding, but for proper testing, I need to export the server pck file once again using the new build from #35516 This is probably a silly question then, but is there a place I can get an export template for this build? Apologies if I am misunderstanding something.

EDIT: Also, when trying to compile from #35516 , whenever I pressed an option in the Godot menu, e.g. 'Import Project', the editor would crash. I worked around this by cloning godot-master and changing the changes files that you changed in #35516 on my local machine engine files, and then recompiling again. Not sure if that was the correct thing to do, but I was wondering if you knew of a fix for that?

[Calinou]

This is probably a silly question then, but is there a place I can get an export template for this build?

To test the pull request, you need to clone @Faless' fork of Godot and switch to the branch that was submitted as a pull request:

git clone https://github.com/Faless/godot.git -b ws/fix_latency_32
cd godot

Then compile Godot for the server platform, following the same instructions as in Compiling for X11:

# -j$(nproc) uses all CPU threads available on the system for building.
scons platform=server tools=no -j$(nproc)

The binary will be located in the repository's bin/ folder.

[notmushroomoverhere]

@Calinou Thank you for the instructions. I have been attempting to use scons, but upon running scons platform=list, my only options are Windows and Javascript; there is no 'server' option nor 'x11'. Hence, attempting the scons command you wrote is not working for me - perhaps it's because I am using Windows?

I'm going to try to use a Linux Virtual Machine to compile the export template instead. Does that sound like a right thing to do? It looks as if the compilation process, from the docs link you sent, is designed to be used/easier to use within Linux (from me looking at 'Distro-specific one-liners')

EDIT: Compiling via the Linux Virtual Machine just crashed - I don't think I'm going to be able to compile that way, but I can try again. I would ideally like to just compile on Windows to the server platform, but I realise that may not be possible. I will try anything to aid testing.

[akien-mga]

@notmushroomoverhere You can build for Linux/Server from Windows Subsystem for Linux: https://docs.microsoft.com/en-us/windows/wsl/install-win10

Depending on the distribution you choose, you can then use the documented oneliners and command line instructions.

But to simplify the process I'm making you test builds for Linux server and Windows editor for #35516.

[akien-mga]

But to simplify the process I'm making you test builds for Linux server and Windows editor for #35516.

Here it is: https://downloads.tuxfamily.org/godotengine/3.2/tmp-test/3.2-rc-test-35516.zip

I'm not sure how you export the PCK for the server build usually, but you might have to trick the export dialog by using the Windows editor build as custom Windows Desktop template, and use that to export a PCK (which should be compatible with the server binary).

[notmushroomoverhere]

@akien-mga I'm a little confused to be honest. I've downloaded the builds you made for me (thanks for those!) Within those files, I'm assuming that I use the windows exectuable, 'godot.windows.opt.tools.64', to run my clients in the editor on. So then, with regards to the 'godot_server.x11.opt' file, am I right in thinking that that is the export template to use to compile the .pck for my Linux server?

If so, I've done that successfully, but I am still experiencing visible latency. I tried:

• Opening the 'godot.windows.opt.tools.64' and running 2 clients.
• Opening 'godot.windows.opt.tools.64' again, and using the 'godot_server.x11.opt' as the export template, exported the server project as a .pck.
• Running the pck like normal in my VM, using 'Godot_v3.2-rc2_linux_server.64' and having clients connect to it.

I'm guessing I've most likely done something incorrectly. In my Linux Virtual Machine, what am I meant to use as the Linux server? Currently, I have been using 'Godot_v3.2-rc2_linux_server.64', which I found here: https://downloads.tuxfamily.org/godotengine/3.2/rc2/
Is there another version that I am meant to use for #35516 ?

[Faless]

@notmushroomoverhere you need to use the godot_server.x11.opt.64 in the link by @akien-mga on the VM, not Godot_v3.2-rc2_linux_server.64

[notmushroomoverhere]

@Faless Thanks - I successfully ran the server that akien-mga gave me on my Linux VM, and I had 2 clients from the editor from akien-mga successfully connect to it. However, I still received high latency (no change occurred).

I have a feeling that I may have not built the .pck correctly, using the right export template. May I have some further instructions on how to go about building the .pck? In a typical stable build, I would just go to Export -> Export as pck/ZIP and use the official export template. What am I meant to use with the new Godot build for the Linux export template?

I also just want to confirm if the Godot client from @akien-mga, 'godot.windows.opt.tools.64', is the correct editor to use? (i.e. is it the build at #35516 ?)

[notmushroomoverhere]

I tried using RC4 with the RC4 export templates and an RC4 Linux server, as well as running the clients out of the RC4 editor. (RC4 has the changes that Faless made in #35516) The latency is still noticeably bad, both when the clients are in the editor, and when the clients have been exported to HTML5. I believe there to have been no noticeable change.

[MarshallJDem]

I can confirm I am also experiencing this latency with my own separate project using RC4. I am running my server off an AWS Ubuntu 18 server, and have tried the following conditions, all of which with the latency:

• Hosting WSS over a self signed certificate
• Hosting WSS over an official CA (Let's Encrypt) certificate
• Hosting in various regions of US and connecting from various regions
• Running the client off my apache server (HTML5 export)
• Running the client locally (HTML5 export)
• Running the client in editor (HTML5)
• Running the client in editor (Mac OSX)