Steamcmd Segmentation Fault on Subsequent Container Runs

[konstander]

Describe the Bug I have been experiencing container error when starting/restarting. The process is cyclical. Restarting the container or creating a new one from the image does not help. I run the image on the synology NAS.

A fragment of the log from the console:

Connecting anonymously to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
 Update state (0x5) verifying install, progress: 0.15 (10493888 / 7124928656)
 Update state (0x5) verifying install, progress: 11.45 (815645455 / 7124928656)
 Update state (0x5) verifying install, progress: 23.10 (1646075198 / 7124928656)
 Update state (0x5) verifying install, progress: 34.57 (2463010896 / 7124928656)
 Update state (0x5) verifying install, progress: 46.11 (3285635716 / 7124928656)
 Update state (0x5) verifying install, progress: 57.92 (4126688358 / 7124928656)
 Update state (0x5) verifying install, progress: 69.68 (4964844867 / 7124928656)
 Update state (0x5) verifying install, progress: 81.42 (5801249014 / 7124928656)
 Update state (0x5) verifying install, progress: 93.14 (6636469831 / 7124928656)
Success! App '1690800' fully installed.
/home/steam/.steam/steamcmd/steamcmd.sh: line 39:    50 Segmentation fault      (core dumped) $DEBUGGER "$STEAMEXE" "$@"
Checking available memory...27GB detected
Setting autosave number to 9
Setting crash reporting to True
Setting max objects number to 2162688
Setting max tick rate to 30
Setting timeout number to 30
Setting max players to 4
Setting autosave interval to 600s
Setting disable seasonal events to 0
Setting network quality number to 3
Setting auto pause to True
Setting autosave on disconnect to True
Checking available storage...8506GB detected
Downloading the latest version of the game...
Redirecting stderr to '/home/steam/.steam/logs/stderr.txt'
Looks like steam didn't shutdown cleanly, scheduling immediate update check
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1687387651
-- type 'quit' to exit --
Loading Steam API...dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
dlmopen libSDL3.so.0 failed: libSDL3.so.0: cannot open shared object file: No such file or directory
OK

Connecting anonymously to Steam Public...OK
...
...

Your Runtime Command or Docker Compose File

{
   "CapAdd" : [],
   "CapDrop" : [],
   "cmd" : "",
   "cpu_priority" : 90,
   "enable_publish_all_ports" : false,
   "enable_restart_policy" : false,
   "enable_service_portal" : null,
   "enabled" : true,
   "env_variables" : [
      {
         "key" : "PATH",
         "value" : "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      },
      {
         "key" : "USER",
         "value" : "root"
      },
      {
         "key" : "HOME",
         "value" : "/root"
      },
      {
         "key" : "LANG",
         "value" : "en_US.UTF-8"
      },
      {
         "key" : "LANGUAGE",
         "value" : "en_US:en"
      },
      {
         "key" : "AUTOPAUSE",
         "value" : "true"
      },
      {
         "key" : "AUTOSAVEINTERVAL",
         "value" : "600"
      },
      {
         "key" : "AUTOSAVENUM",
         "value" : "5"
      },
      {
         "key" : "AUTOSAVEONDISCONNECT",
         "value" : "true"
      },
      {
         "key" : "CRASHREPORT",
         "value" : "true"
      },
      {
         "key" : "DEBUG",
         "value" : "false"
      },
      {
         "key" : "DISABLESEASONALEVENTS",
         "value" : "false"
      },
      {
         "key" : "GAMECONFIGDIR",
         "value" : "/config/gamefiles/FactoryGame/Saved"
      },
      {
         "key" : "GAMESAVESDIR",
         "value" : "/home/steam/.config/Epic/FactoryGame/Saved/SaveGames"
      },
      {
         "key" : "MAXOBJECTS",
         "value" : "2162688"
      },
      {
         "key" : "MAXPLAYERS",
         "value" : "4"
      },
      {
         "key" : "MAXTICKRATE",
         "value" : "30"
      },
      {
         "key" : "NETWORKQUALITY",
         "value" : "3"
      },
      {
         "key" : "PGID",
         "value" : "1000"
      },
      {
         "key" : "PUID",
         "value" : "1000"
      },
      {
         "key" : "SERVERBEACONPORT",
         "value" : "15000"
      },
      {
         "key" : "SERVERGAMEPORT",
         "value" : "7777"
      },
      {
         "key" : "SERVERIP",
         "value" : "0.0.0.0"
      },
      {
         "key" : "SERVERQUERYPORT",
         "value" : "15777"
      },
      {
         "key" : "SKIPUPDATE",
         "value" : "false"
      },
      {
         "key" : "STEAMAPPID",
         "value" : "1690800"
      },
      {
         "key" : "STEAMBETA",
         "value" : "false"
      },
      {
         "key" : "TIMEOUT",
         "value" : "30"
      }
   ],
   "exporting" : false,
   "id" : "90d43a03b0c4a643eeee825838cb6b9dfeda683d8e73ef1ba30786d0b89b1735",
   "image" : "wolveix/satisfactory-server:latest",
   "is_ddsm" : false,
   "is_package" : false,
   "links" : [],
   "memory_limit" : 17179869184,
   "name" : "wolveix-satisfactory-152",
   "network" : [
      {
         "driver" : "bridge",
         "name" : "satisfactory"
      }
   ],
   "network_mode" : "satisfactory",
   "port_bindings" : [
      {
         "container_port" : 15000,
         "host_port" : 0,
         "type" : "udp"
      },
      {
         "container_port" : 15777,
         "host_port" : 0,
         "type" : "udp"
      },
      {
         "container_port" : 7777,
         "host_port" : 0,
         "type" : "udp"
      }
   ],
   "privileged" : false,
   "shortcut" : {
      "enable_shortcut" : false,
      "enable_status_page" : false,
      "enable_web_page" : false,
      "web_page_url" : ""
   },
   "use_host_network" : false,
   "volume_bindings" : [
      {
         "host_volume_file" : "/docker/satisfactory",
         "mount_point" : "/config",
         "type" : "rw"
      }
   ]
}

System Specs:

===== START ISSUE REPORT =====
OS:  Linux DS923plus 4.4.180+ #42962 SMP Fri May 26 23:30:43 CST 2023 x86_64 GNU/Linux synology_r1000_923+
CPU:
RAM: 27GB/32GB
HDD sys: 1GB/2GB (79% used)
HDD total: 11T/2.2T (21% used)
===== END ISSUE REPORT =====

Additional Context

docker images | grep satisfactory
wolveix/satisfactory-server   latest    e2494126d956   9 days ago    413MB

[wolveix]

Ah, Synology. Thanks for opening this @konstander. Is that RAM calculation without the server running? If so, that's not enough RAM to run the server.

This looks to be the same error as experienced here: https://github.com/parkervcp/eggs/issues/2302

It's not an issue with the image, but with steamCMD. I can't properly look into it right this second, but I can take a proper look tomorrow

[konstander]

Ah, Synology. Thanks for opening this @konstander. Is that RAM calculation without the server running? If so, that's not enough RAM to run the server.

This looks to be the same error as experienced here: parkervcp/eggs#2302

It's not an issue with the image, but with steamCMD. I can't properly look into it right this second, but I can take a proper look tomorrow

27Gb of free RAM out of 32Gb. The container is limited in settings to 16GB. I have read the link above carefully. Apparently you are right about something with SteamCMD (after recent steam updates) and SDL lib v3.

Ok. Thx.

[konstander]

The SDL team is focusing on SDL 3.0 and all new feature development will be happening there.

https://github.com/libsdl-org/SDL/releases Only 2.x version now. :(

[konstander]

Hey, @wolveix Perhaps this can help in correcting the image? https://github.com/libsdl-org/SDL/issues/7841

steamcmd doesn't actually use SDL, it's being loaded by the Steam code as an optional path, but it's not necessary...

... ... Placing the 32 bit libSDL3.so.0 and libSDL3.so.0.0.0 file in the steamcmd/linux32 folder fixt steamcmd complaining about them.

[wolveix]

@konstander I've just verified that this is happening for me too, due to the SteamCMD update you referenced. However, it doesn't stop the container at all for me. Is it causing your container to not properly startup?

[konstander]

@wolveix Based on the text in the log, there is a file integrity check, then the libSDL 3 error and the game server does not start as usual. The container stops.

[wolveix]

@konstander hm, that's weird. Maybe mine still starts since I already have the game files downloaded

[wolveix]

@konstander I tried with a clean directory, and it still continued. I'm not sure why your container is terminating prematurely. I'll test a fix with the SDL files present, but that may not be the issue.

Oh wait, I've just spotted the segmentation fault error in your log. That'll be the real issue.

[wolveix]

@konstander I don't personally have experience with Synology's interface, but it's been the cause of a lot of issues since the inception of this repo.

The spec you gave in place of a normal Docker run or Docker Compose file is interesting, that's what Synology gives you, right? What did you specify when initially running the container?

[konstander]

@wolveix

Oh wait, I've just spotted the segmentation fault error in your log. That'll be the real issue.

Perhaps. All I understood was that the files are checked 100%, similarly, the files are downloaded 100% (if you delete everything). But then the server does not start. Perhaps some error during verification or transmission of an "error signal" to the log stops the process. I don't understand well at this moment.

It usually ends there: home/steam/.steam/steamcmd/steamcmd.sh: line 39: 50 Segmentation fault (core dumped) $DEBUGGER "$STEAMEXE" "$@"

The spec you gave in place of a normal Docker run or Docker Compose file is interesting, that's what Synology gives you, right? What did you specify when initially running the container?

I am using the docker application (package) in synology nas. The settings file I have given is the container configuration export file. Unfortunately, input options via the "console" (as on a PC) via the DSM synology interface (GUI) are not supported, I did not try using SSH. I repeat, earlier everything worked with your image. All container parameters are perfectly configured in the graphical shell of the docker application.

[konstander]

@wolveix Have you updated the image? Noticed the image was updated a few hours ago. I have now re-deleted everything and re-downloaded your image and made a new container based on it. After the launch, some kind of update was downloaded... Everything started! but... I had to stop the container and restart it. Got the "segmentation" error above again. :(

[konstander]

An interesting observation: 1) I delete the file in the config folder..\gamefiles\Factory Server.sh The container starts properly! 2) In the game, I make the primary server settings - the administrator's name, password, etc. I try to load the save - the container crashes. This save file in single mode works. No mods are installed.

It was possible to load the server several times successfully with a save. Deleted the file every time FactoryServer.sh (yes, in the process of launching it, it was created anew and everything seemed to be going as it should).

Error again all in a circle, ohh God... home/steam/.steam/steamcmd/steamcmd.sh: line 39: 50 Segmentation fault (core dumped) $DEBUGGER "$STEAMEXE" "$@"

[konstander]

@wolveix Some more of my research in the process of layout errors. In the container settings, I set the value SKIP UPDATE = true Starting, stopping, restarting the container goes fine. Very interesting...

If I understood everything correctly, SteamCMD incorrectly transmits information about the results of file verification to the log, something "listens", does not receive the desired response... Somewhere I read about a similar error, I find it difficult to remember where exactly.

[wolveix]

Yeah, setting SKIPUPDATE=true will completely ignore SteamCMD, hence why it works :D I still don't really understand why your specific environment isn't handling SteamCMD properly, and crashing. I did update the image manually last night, but I merely rebuilt it in the hopes that an upstream update may resolve the issue for you.

I really appreciate your hard work investigating this issue, and wish that I had more to add. I'm pretty slammed with work right now, but I'm checking in when I can :)

[konstander]

Thank you. I will use this temporary solution for now. I hope when you have more time you will pay attention to this problem more closely.

[m3talliz3d]

@wolveix Some more of my research in the process of layout errors. In the container settings, I set the value SKIP UPDATE = true Starting, stopping, restarting the container goes fine. Very interesting...

If I understood everything correctly, SteamCMD incorrectly transmits information about the results of file verification to the log, something "listens", does not receive the desired response... Somewhere I read about a similar error, I find it difficult to remember where exactly.

@konstander I had the same issue and when I stopped set SKIPUPDATE = true it worked fine.

I will try to investigate this behavior...

@wolveix If I want to debug should I use the docker DEBUG command or or use the DEBUG value mentioned in the EnviromentalVariable?

[wolveix]

@m3talliz3d use Docker's debug, since the DEBUG environment just prints some additional information for troubleshooting issues. It's not particularly applicable in this use-case. Are you also using Synology?

[m3talliz3d]

@wolveix I am using satisfactory Experimental as Docker container on Ubuntu VM that is hosted on ESXi, also I got another one on Proxmox CT but not using it as it has less resources.

[wolveix]

@m3talliz3d and you're experiencing the same issue with the game server not launching?

[m3talliz3d]

@wolveix well, when I choose update to be true, it show an error with network when trying to connect and fail to connect to game.

Yet, I need to confirm further and perform another test, since I already decreased the resource limits (reserve 6GB and limit to 16GB [My VM was 18GB) it was failing, and when I reverted back to set update=false and increased VM Ram to 24 and limits to the one you provided (doc compose) it worked fine.

Anyway, I will test again when am back home different scenarios just to isolate the culprit of the issue and will report back.

I would like to thank you for your efforts in dockerizing the game server. Fantastic work.

[silen72]

I had the same or at least a very similar issue today after switching to experimental. On startup it did a suspicously small download (like 90MB), then complained about libSDL3.so.0 and crashed with a seg fault shortly after. In between it said "no executable found". Rinse and repeat. After switching back to early access, it downloaded a considerably larger package, installed it and the server is back to functional. Not compatible with my satisfactory client but functional. So experimental = broken, EA = working - at least for me. Not on synology but on a home made server running ubuntu server 22.04.02.

[wolveix]

@silen72 ah, no, "no executable found" is most likely due to you running an older image (see #187). Please make sure you're running the latest image, as Update 8 required a slight change to the image to pickup the new server executable location. Your image hash should look like this:

❯ docker images | grep satisfactory
wolveix/satisfactory-server                 latest              f13f51f7abdf   2 days ago      417MB

[wolveix]

@konstander I just realised that this whole issue is a duplicate of #188. In that, @Reticulmz resolved their issue by reinstalling Docker. Of course I'd love to gain a greater insight into what may be causing this issue, but I do suspect that it's not an issue with the image itself

[Reticulmz]

In my case, I think it's because wsl2 didn't free memory and steamcmd didn't start due to lack of memory. I think you can avoid the error by specifying the memory limit or swap in .wslconfig for example.

[wolveix]

@Reticulmz ah, I really appreciate that insight :)

[Reticulmz]

I've been doing some digging around, and maybe it's due to running out of stack space. Try checking with ulimit -a.

[m3talliz3d]

That is strange... everything is working fine now.

===== START ISSUE REPORT ===== Hardware: ESXi 6.7U3 OS: Linux satisfactoryd 5.4.0-153-generic #170-Ubuntu (Ubuntu 20.04.1 LTS) SMP Fri Jun 16 13:43:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz RAM: 10GB/11GB HDD: 19GB/98GB (20% used) ===== END ISSUE REPORT =====

Note: RAM = 23GB/24GB

T#1: (update=false) -> Ran docker compose up . -> connected to the game and it worked.

T#2: -> Stopped container with docker compose down . -> reverted update to true as default" . -> Started container. -> Connected to game successfully.

T#3: -> Stopped container with docker compose down . -> Set limits to --> Limit = 10 GB --> Reservation = 8GB -> Started container. -> Connected to game successfully.

T#4: -> Stopped container with docker compose down . -> Power off VM and change its RAM to 12GB instead of 24GB -> Started container. (with same docker-compose changes in T#3) -> Connected to game successfully.

I am not sure what was the issue exactly before but the error I had previously was

[2023.06.29-08.52.31:353][980]LogNet: NetworkFailure: ConnectionTimeout, Error: 'UNetConnection::Tick: Connection TIMED OUT. Closing connection.. Elapsed: 30.00, Real: 30.00, Good: 30.00, DriverTime: 68083.64, Threshold: 30.00, [UNetConnection] RemoteAddr: 172.18.0.1:60447, Name: IpConnection_2147481141, Driver: GameNetDriver NetDriverEOSBase_2147481943, IsServer: YES, PC: NULL, Owner: NULL, UniqueId: EOS:(EOS)3f9f7a418b434065adb3853ff12de642|000289f1418a4c7dbf726ddc36c37f89'

now it looks like everything is running smoothly.

error regarding libSDL3 looks like safe to ignore, it didn\t cause any issues on my side

satisfactory-server  | Loading Steam API...dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
satisfactory-server  | dlmopen libSDL3.so.0 failed: libSDL3.so.0: cannot open shared object file: No such file or directory
satisfactory-server  | OK

@Reticulmz What is your RAM available to be used or the one that WSL use/allocate?

[silen72]

@wolveix Thanks for the hint, that did the trick for me.

[konstander]

@wolveix

@konstander I just realised that this whole issue is a duplicate of #188. In that, @Reticulmz resolved their issue by reinstalling Docker. Of course I'd love to gain a greater insight into what may be causing this issue, but I do suspect that it's not an issue with the image itself

Sounds like critical luck. ^^ The solution is more suitable for a PC. Of course I'm trying to reinstall the docker package on my synology NAS. I'll tell you what happened. Most likely, the problem is really not in the image itself, but in steamcmd.

[konstander]

Checked reinstalled docker. At the same time, I installed the new DSM update DSM 7.2-64570 Update 1 recently, a new version has arrived. With the SKIP UPDATE = true the server starts. With the SKIP UPDATE = false restarting the container results in a fragmentation error as usual: home/steam/.steam/steamcmd/steamcmd.sh: line 39: 50 Segmentation fault (core dumped) $DEBUGGER "$STEAMEXE" "$@" Ideas are over for now.

[m3talliz3d]

@konstander are you able to get the core dump file name?

It should be in /core as far as I recall, and if that is the case, do a mount for /core to persistent in case it is huge dump.

[konstander]

@m3talliz3d

@konstander are you able to get the core dump file name?

It should be in /core as far as I recall, and if that is the case, do a mount for /core to persistent in case it is huge dump.

What do you mean, describe in more detail? Tried to mount /core nothing. What exactly is the path in the container (if we take the /config directory as an example) to mount?

[m3talliz3d]

As far as I recall, segmentation fault could be caused because process ran out of memory, thus it created a core dump, core dump in Linux is a way where...... when a process fail, it dumps its memory data ito a file so it can inspected later with specific tool.

Core dump is usually saved in location '/core', in case this is the scenario then you could mount './core:/core' to have it persistent.

I am not sure what is the issue, but as Ex-VMware support we used to have our core dumps in /core directory.

Got lost? @konstander

[konstander]

@m3talliz3d It's clear, but I don't know in which directory the dump is saved in this particular image. Maybe it will tell me @wolveix

[m3talliz3d]

@konstander when it show this error. Does the container stop? If not can you exec into container and perform 'df -h' and check if you can notice which partition/directory got almost full.

I will check the file structure when I get home.

[konstander]

@m3talliz3d df -h

# docker exec -ti satisfactory-server sh
# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/cachedev_1   11T  2.3T  8.2T  22% /
tmpfs                    64M     0   64M   0% /dev
tmpfs                    16G     0   16G   0% /sys/fs/cgroup
shm                      64M     0   64M   0% /dev/shm
/dev/mapper/cachedev_1   11T  2.3T  8.2T  22% /core
/dev/mapper/cachedev_1   11T  2.3T  8.2T  22% /etc/hosts
tmpfs                    16G     0   16G   0% /proc/acpi
tmpfs                    16G     0   16G   0% /proc/scsi
tmpfs                    16G     0   16G   0% /sys/firmware

[m3talliz3d]

@konstander ls /core

[konstander]

@m3talliz3d nothing... The directory is empty. So I mounted /core correctly earlier, there are just no dumps there.

[konstander]

For the sake of interest, I tried to launch an image of another author, check what will happen. You know what? Starts, stops restarts and no fragmentation errors. >< p.s. By the way, thanks to the update of the DSM interface in synology, the docker package has acquired the function of "launching projects", including through the Docker Compose file. It is a pity that this did not solve the fragmentation problem.

[m3talliz3d]

That is such a fantastic news, I hope you will still consider trying with wolveix image until a resolution is found.

I am actually very interested to get that issue fixed and if applicable I don't mind having a session with you, me and @wolveix to troubleshoot live.

Thanks for your update!

[wolveix]

@konstander would you mind sharing what image ended up working for you? :)

[konstander]

@konstander would you mind sharing what image ended up working for you? :)

@wolveix Sure, url The author has problems with access rights "out of the box" (the container does not have write/edit rights to the mounted directory), but it doesn't matter. I hope we will find a solution together :) my setup here:

version: '3'
services:
    satisfactory-server-01:
        container_name: 'satisfactory-server-01'
        hostname: 'satisfactory-server-01'
        image: 'donimax/satisfactory-docker:latest'
        ports:
            - '7777:7777/udp'
            - '15000:15000/udp'
            - '15777:15777/udp'
        volumes:
            - '/volume1/docker/test-srv:/config'
        environment:
            - MAXPLAYERS=4
            - PGID=1000
            - PUID=1000
            - STEAMBETA=false
            - SERVERBEACONPORT=15000
            - SERVERGAMEPORT=7777
            - SERVERIP=0.0.0.0
            - SERVERQUERYPORT=15777
            - SKIPUPDATE=false
            - STEAMAPPID=1690800
        restart: unless-stopped
        deploy:
          resources:
            limits:
              memory: 16G
            reservations:
              memory: 12G

[wolveix]

@konstander huh, that's interesting. Looks like they forked my repo to begin with (everything is laid out identically to an old commit), but they never adopted our permissions fixes.

That could potentially be where the issue lies? I'll have a proper look through in a bit, but that's really interesting 😅

[wolveix]

It's hard to remember everything that we've tried during this troubleshooting :P Have you tried running without a bind mount? So removing the volumes: declaration in your Docker Compose

[konstander]

No. I will try.

[comzyh]

Same issue after I upgraded from last year's version eee7de98370e to f13f51f7abdf without cleanup. I will try to debug.

[comzyh]

My problem just disappears.

For who want to debug, can just try to use docker-compose run to avoid the container to be destroyed after the server fail.

docker-compose run --entrypoint /bin/bash --service-ports -- satisfactory-server

then manually use /init.sh to start the server.

I use /init.sh to start the server and the first try failed as described in this issue and the second run is just a success.

Also I can confirm that the issue is not related to lib libSDL3.so

libSDL3.so.0 failed: libSDL3.so.0: cannot open shared object file: No such file or directory

I got this libSDL3 line output still, but my server is already working.

[wolveix]

@konstander any update? :)

[konstander]

I'm sorry, I'm sick. I'll resume testing as soon as I feel better.

[konstander]

Still, I did a short test. I used the standard Docker Compose file recommended by you. Created a container without mounting the "config" folder. Removed the line: `volumes:

• '/path/to/config:/config' With theSKIP UPDATE = false Restarting the container, I get a fragmentation error. home/steam/.steam/steamcmd/steamcmd.sh: line 39: 50 Segmentation fault (core dumped) $DEBUGGER "$STEAMEXE" "$@"`