Server RAM Usage and Client Connection Timeout

[IMarkoMC]

Describe the Bug When i try joining the game, the server ram jumps from ~2GB to ~7GB (It used less than 5gb earlier) The server gives a lot of errors like

[2022.09.20-23.25.55:491][784]LogNavigationDirtyArea: Warning: Skipping dirty area creation because of empty bounds (object: FoliageInstancedStaticMeshComponent /Game/FactoryGame/Map/GameLevel01/Tile_X5_Y1.Tile_X5_Y1:PersistentLevel.InstancedFoliageActor_0.FoliageInstancedStaticMeshComponent_283)

The client gives a timeout error while those errors are happening.

Your Runtime Command or Docker Compose File Started it with the command provided in this repo to rule out some things

docker run -d --name=satisfactory-server -h satisfactory-server -e MAXPLAYERS=4 -e PGID=1000 -e PUID=1000 -e STEAMBETA=false -v /path/to/config:/config -p 7777:7777/udp -p 15000:15000/udp -p 15777:15777/udp wolveix/satisfactory-server:latest

System Specs (please complete the following information):

===== START ISSUE REPORT =====
OS:  Linux ubuntu 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
CPU: Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz
RAM: 10GB/11GB
HDD: 33GB/109GB (32% used)
===== END ISSUE REPORT =====

Additional Context Entire server log

[silentnyte]

I am having similar issues using the truecharts helm instance.

[Kadigan]

Could you check if you can connect after the server has actually finished loading the game?

[IMarkoMC]

Could you check if you can connect after the server has actually finished loading the game?

You can connect once it gives all of the errors and the memory stops going up but it runs weird. Combat is broken (movement is glitched https://streamable.com/b97n7g ) and the game itself feels weird

I'm in the same network as the server is, that's not a weird networking issue between a remote server and the client

[MattsBos]

So, you are now connected fine as you are in game?

The glitchy thing is a new bug introduced, will have to be fixed by Satisfactory, that happens with the spitters for everyone this update.

Also, the mention of RAM is as designed, Satisfactory Update 6 dedicated server requires 12GB of RAM.

[IMarkoMC]

Also, the mention of RAM is as designed, Satisfactory Update 6 dedicated server requires 12GB of RAM.

12GB of ram a satisfactory server? For 2 players no way It ran flawlessly with 6gb earlier on that's 2x the memory.

I guess I'll stop playing the game until the memory is reduced lol

So, you are now connected fine as you are in game?

If you wait the 5 minutes it takes to load and give all that errors it kinda works. It's not the same as it worked before, the game runs weird even with enemies not targeting you properly. That might be because it's nearly out of memory but i refuse to give a single server 12G of ram for 2 users and one world

[MattsBos]

Yeah, it's quite a big change, but it's been introduced ever since Update 6 hit experimental. 12GB is the new requirement. My server is currently using 11.4GB and it's just me in there. I don't think player count matters as it stays roughly the same with 4 players active.

[IMarkoMC]

Well time to shut down the server, get a way to downgrade to update 5 and wait until the memory usage goes back to something normal aka <6GB

Even Minecraft eats less memory with more users

[wolveix]

Yeah, it sounds like a memory leak or a bug introduced with the update. Just wait until it's resolved :) Not a lot we can do in the meantime unfortunately!

[Kadigan]

You can connect once it gives all of the errors and the memory stops going

So - user error: you tried connecting before the server was ready to accept connections.

Combat is broken (movement is glitched https://streamable.com/b97n7g )

My suspicion is that this is due to the projectile the spitter throws at you. The effect is similar to going through other world objects, and the bug is similar to the whole boulder & hatcher issue (where you'd get weird jerking around when you go through areas these things used to be in, but no longer are). Regardless, that's a game issue, not a networking/connectivity issue.

and the game itself feels weird

... I'm not sure how to understand that one, to be honest.

I'm in the same network as the server is, that's not a weird networking issue between a remote server and the client

You're correct, it's not.

[IMarkoMC]

Kinda sad seeing that in all this time nothing changed in how they handle updates. Update news: Added new bugs and new memory leaks

[IMarkoMC]

and the game itself feels weird ... I'm not sure how to understand that one, to be honest.

From update 5 to update 6 things just doesn't work right. Some enemies are dumb and they just sit there from time to time (They target the user for a moment and then they sit still), objects in the belts stop moving more often

[Kadigan]

get a way to downgrade to update 5

As far as I'm aware, that's not something that's been made available. You would need to fetch an older codebase, for both client and server.

wait until the memory usage goes back to something normal aka <6GB

Not sure where you get those numbers - my EA Update 5 ran between 6.7 and 7.8 GiB (Linux x64).

Regardless: the developers are aware that the server requires more RAM, and they say it's related to all of the changes that have been made to it over the course of Update 6 development. While it will likely go down somewhat, I honestly do not expect it will go below 9 GiB. Considering that the game is still in development, and how RAM-hungry UE games are - I rather expect it to go up.

Even Minecraft eats less memory with more users

That may be so. This isn't Minecraft, however. HLDS used 50 MiB of RAM, you know? I wish we could go back to that.

[Kadigan]

Some enemies are dumb and they just sit there from time to time (They target the user for a moment and then they sit still)

Update 6 introduced a substantial rewrite of enemy AI. I suspect it's related to that, and will see future work.

[IMarkoMC]

Not sure where you get those numbers - my EA Update 5 ran between 6.7 and 7.8 GiB (Linux x64).

I was playing with 2 friends less than a week ago and it was using 5 - 5.7gb (capped at that 6GB)

[Kadigan]

I was playing with 2 friends less than a week ago and it was using 5 - 5.7gb (capped at that 6GB)

Where did you get those numbers? Can you provide a command?

And I assume if it goes beyond the cap, it'll just OOM?

[IMarkoMC]

Where did you get those numbers? Can you provide a command?

There's a public Pterodactyl panel egg this one. I was using that one before all of this happened

And I assume if it goes beyond the cap, it'll just OOM?

Never went OOM for me in more than a month

[IMarkoMC]

I honestly do not expect it will go below 9 GiB. Considering that the game is still in development, and how RAM-hungry UE games are - I rather expect it to go up.

If using 9+GiB for a game server for the developers of the game is ""normal"" we have some big issues going forward

[Kadigan]

There's a public Pterodactyl panel egg

Unfortunately, I have no clue how to use that. Can you tell me where Pterodactyl gets this memory reading from? Because, like I said, I'm getting completely different readings from basic tools like top/htop, ps etc.

[IMarkoMC]

Unfortunately, I have no clue how to use that. Can you tell me where Pterodactyl gets this memory reading from? Because, like I said, I'm getting completely different readings from basic tools like top/htop, ps etc.

HTOP reported the same memory usage as the panel i wish i had some screenshot but i dont

[Kadigan]

If using 9+GiB for a game server for the developers of the game is ""normal"" we have some big issues going forward

Sadly, this is the world we now live in. ARK servers with a number of mods loaded in range from 13 to 18 GiB of RAM, and it doesn't look like this will go down any time soon (ARK is another example of a UE game). Similarly, CONAN Exiles - about 12-14 GiB (another UE game).

I happen to agree with you, but there's nothing I can do - except not buying and playing the games, I suppose.

[Kadigan]

HTOP reported the same memory usage as the panel i wish i had some screenshot but i dont

Which value are you using as a reference?

[IMarkoMC]

I'll just stop playing the game again until they get the memory usage to a reasonable number. Well, time to wait i guess

[Kadigan]

I'll just stop playing the game again until they get the memory usage to a reasonable number. Well, time to wait i guess

Like I said - I fully expect the requirement to - if not grow, then at least remain basically where it is. So, I assume you're done with the game, on principle.

[IMarkoMC]

Found a video from a few days ago. 2 players online

[Kadigan]

Yeah, that doesn't really help, as there's a number of performance metrics it might be referencing. I'm sourcing my information from the "resident memory" metric of the executable. Where is Pterodactyl sourcing it, that it can display a lower value?

And how does it report 6.44 GB of disk, when I know for a fact that the game takes up a little over 7 GiB?

[IMarkoMC]

Yeah, that doesn't really help, as there's a number of performance metrics it might be referencing. I'm sourcing my information from the "resident memory" metric of the executable. Where is Pterodactyl sourcing it, that it can display a lower value?

If you want specifics about that, the developers of the panel should be able to answer that. It's the same reported in docker stats but idk from where they take it

And how does it report 6.44 GB of disk, when I know for a fact that the game takes up a little over 7 GiB?

The folder now is 7.0G according to du -sh and it's reporting 6.95 GB. Not something that worries me

[Kadigan]

7.65 GiB

Now that is much more in-line with what I've been seeing. And since Docker doesn't actually spin up a separate OS, it should be limited mostly to what the application is using.

So, unless this was a report from a different point in time (or a different server version), that 5.41 GiB reported memory use is bogus.

[IMarkoMC]

Now that is much more in-line with what I've been seeing. And since Docker doesn't actually spin up a separate OS, it should be limited mostly to what the application is using.

That's the memory that it's using now.

The 5.4GB was the 17 of September. Anyways this is not going anywhere

[Kadigan]

Anyways this is not going anywhere

You're right. I went ahead and asked in Pterodactyl Discord - they should be qualified to clear up any misunderstandings I have.

[Kadigan]

At any rate, I propose this be closed as user error/not a bug.

[IMarkoMC]

At any rate, I propose this be closed as user error/not a bug.

It's a game bug, not a user error

[Kadigan]

It's a game bug, not a user error

The fact that you tried connecting before the server was ready to accept connections is user error.

Every other aspect of the reported issue is unrelated to the reported problem.

[IMarkoMC]

The fact that you tried connecting before the server was ready to accept connections is user error. Every other aspect of the reported issue is unrelated to the reported problem.

I didn't try connecting before the server was ready to accept connections, i can leave it for 15 minutes and it will do nothing until i try connecting.

Anyways this is pointless

[wolveix]

@IMarkoMC to clarify, are you still experiencing the issue? I'd like to keep this issue open if so, as I suspect others may try to open related issues.

[wolveix]

Also, lets keep things civil please guys :) If the game server was accepting connections prior to the save being loaded, that is a design flaw mixed with user error in my opinion. If the issue is still happening regardless of how much time has been left, as @IMarkoMC is reporting, then it's clear that there is in fact a bug with the game server itself.

[wolveix]

Also, the mention of RAM is as designed, Satisfactory Update 6 dedicated server requires 12GB of RAM.

@MattsBos that's interesting, I wonder when/why they changed that? The game server (update 4 & 5) needed half of that.

[wolveix]

@IMarkoMC @silentnyte @Kadigan @MattsBos I've pushed a commit which checks the server's available RAM on startup to be in-line with the server's new minimum RAM requirements.

[IMarkoMC]

to clarify, are you still experiencing the issue? I'd like to keep this issue open if so, as I suspect others may try to open related issues.

Yes. The server is still loading the world once you try connecting and the client timeouts.

I just restarted it to check. Started the server waited 5 minutes until joining (no logs meanwhile) and the same thing happens

[wolveix]

@IMarkoMC due to the new RAM requirements, I believe this is unfortunately a case of your system not being powerful enough to run the game server. I don't really understand why CSS have doubled the minimum RAM requirement, but unfortunately it's out of my control.

[IMarkoMC]

I left it uncapped. And it's using 8GB, no matter if i limit it or not it's behaving the same

[wolveix]

@IMarkoMC the minimum requirement is 12GB, your system appears to have 11GB.

[IMarkoMC]

@IMarkoMC the minimum requirement is 12GB, your system appears to have 11GB

It's not using nearly 11GB of ram. But let me move it to another server and test it, i don't expect any changes

[wolveix]

@IMarkoMC I appreciate that, but it's the minimum requirement. As you're experiencing issues, it's reasonable to assume that your lack of available memory could be the cause :) The game could be failing to load something, preventing further memory allocation. Please try it on another server as you've suggested and let us know how you get on

[MattsBos]

Also, the mention of RAM is as designed, Satisfactory Update 6 dedicated server requires 12GB of RAM.

@MattsBos that's interesting, I wonder when/why they changed that? The game server (update 4 & 5) needed half of that.

The very first release of Update 6 on the experimental branch came with a considerable rewrite and update (even though it looks like a small update) causing the new requirements. Everyone hopes this will be changed/optimized, but considering the stability it brings and how good the server runs now I think it's for the better. RAM is cheap, but the simple hobbyist self-hosts on old systems will have some issues with the higher RAM requirement.

Ever since Update 6, the servers I have running are using 11+GB in memory, regardless of connected players or size of the save game. Limiting this manually will cause issues sooner or later. In hibernation the memory usage is around 3GB max. Soon as 1 player connects it will jump to 11GB.

This very update was rolled out to Early Access just a few days ago, I expect a lot more people to find memory problems.

Also, I don't know why Marko is claiming/seeing such low values and why that would even work on the previous Update 5. As I cannot replicate any of that and don't see people with that low memory in any of the Discord servers (both Pterodactyl and official Satisfactory) I'm in.

To close off, I don't understand how you fetched that ingame footage if you can't connect to your server. Did you just get that from the internet somewhere or did you actually manage to connect to your server eventually, as in your first message you say it doesn't work, then later you say you got in and then again you say it was impossible.

[IMarkoMC]

To close off, I don't understand how you fetched that ingame footage if you can't connect to your server. Did you just get that from the internet somewhere or did you actually manage to connect to your server eventually, as in your first message you say it doesn't work, then later you say you got in and then again you say it was impossible.

if you try connecting once, it kicks timeouts, i waited a few minutes and tried again once the server finished giving errors and i was able to get in

Also, I don't know why Marko is claiming/seeing such low values and why that would even work on the previous Update 5

I have a bunch of friends that hosted satisfactory prior to update 6 and they never used more then 6gb, limiting the ram shouldn't make the game not work as it didn't do earlier on and it's not using 100% of that

[wolveix]

The very first release of Update 6 on the experimental branch came with a considerable rewrite and update (even though it looks like a small update) causing the new requirements. Everyone hopes this will be changed/optimized, but considering the stability it brings and how good the server runs now I think it's for the better. RAM is cheap, but the simple hobbyist self-hosts on old systems will have some issues with the higher RAM requirement.

@MattsBos thank you so much for the information! I haven't been following Update 6 too closely, so I was completely unaware of the internal changes.

I have a bunch of friends that hosted satisfactory prior to update 6 and they never used more then 6gb, limiting the ram shouldn't make the game not work as it didn't do earlier on and it's not using 100% of that

@IMarkoMC right, because pre-Update 6 had a much lower RAM requirement. The behaviour you're experiencing makes sense given the new requirements.

[IMarkoMC]

@IMarkoMC right, because pre-Update 6 had a much lower RAM requirement. The behaviour you're experiencing makes sense given the new requirements.

It's doing the same with 32GB of ram (World not loading until you join, a lot of errors and the client ends up timeouting).

You need to rejoin after that to play

https://www.youtube.com/watch?v=04EeCmPK8MI

Skip to 3:14 if you don't want to see 3 minutes of the server doing nothing.

This doesn't seem to be a "not enough ram issue" as it's behaving the same in a system with 32GB of ram.

Anyways, this is not something that anyone here can fix i already spent way too much time dealing with this, I'll wait and see how things go in the next update

[MattsBos]

I just watched the video, what instantly gets my attention is that the process of starting up the server and then the process of actually loading the map when you connect is taking absolute ages.

I just double checked my times to verify:

• Starting the server takes 12 seconds to complete
• First player connecting to fresh server takes 20 seconds to complete.

Keep in mind that most timers in Satisfactory are set for 30 seconds, that's also what the error tells you in the client, the connection timed out. Judging by your video, it just seems like your server isn't fast enough to load everything within that time window, the server does not send info to the client to keep alive, so the client disconnects as it didn't receive data for 30 seconds. This again explains why the second time it does work, because the server finished loading the map in the meantime.

Your server is the bottleneck and does not have enough performance to be ready within 30 seconds. This makes it all very clear, thanks for the video!

Just a quick visit back to the 11GB RAM if you want to go back to that, your server will use more than that over time, I have never seen it go over 12GB though. But 11GB will fail eventually, even if you get it up and running.

[IMarkoMC]

Keep in mind that most timers in Satisfactory are set for 30 seconds, that's also what the error tells you in the client, the connection timed out. Judging by your video, it just seems like your server isn't fast enough to load everything within that time window, the server does not send info to the client to keep alive, so the client disconnects as it didn't receive data for 30 seconds. This again explains why the second time it does work, because the server finished loading the map in the meantime.

Interesting, has that changed in the update 6? because i didn't had any issues earlier on, it was just turning on the container, waiting for it to load the world (that now doesn't do on startup) and just play without any issues.

Knowing that i might do a last test and move it to a Ryzen 9 3900X with gen4 disks and see how it does there

[MattsBos]

No, there's just more data to load, so this is probably just pushing your over the edge. In Update 5 the memory usage is half of update 6 so you would only need half the time to load the map on the server. Maybe that was just within limits in Update 5.

Look at your video when you join, you can see the server working, loading everything into memory and you disconnect when it's around 6-7GB memory while it has to go to around 10.xGB to be ready and accept players. And that amount would just fit the time window for Update 5, but not for 6.

I actually don't know if Update 5 server loaded the map on startup, I think it also did that on first connect. But I'm not certain.