[Bug]: arma3 crashes when adding Headless Clients

Thadah commented 1 year ago

Panel Version

1.11.3

Wings Version

1.11.6

Service

arma3

Modified

No, I did not modify the egg

Expected Behavior

Server to start normally using Headless Clients.

Actual Behavior

Server crashes (core dumped) when starting with any headless client.

[STARTUP]: Clearing Headless Client profiles cache...
[STARTUP]: Starting 1 Headless Client(s).
[STARTUP]: Headless Client 1 launched.
[STARTUP]: Starting server with the following startup command:
./arma3server_x64 -ip=0.0.0.0 -port=2302 -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="" -serverMod="" "-noLogs"
19:42:22 SteamAPI initialization failed. Steam features won't be accessible!19:42:22 SteamAPI initialization failed. Steam features won't be accessible!
/entrypoint.sh: line 398:    52 Segmentation fault      (core dumped) ${modifiedStartup}
PTDL_CONTAINER_ERR: There was an error while attempting to run the start command.

Steps To Reproduce

Step 1 Install an arma server Step 2 Run with headless client

Install logs

https://pteropaste.com/gfp1w6wyi13s/

lilkingjr1 commented 1 year ago

I just tried it myself and couldn't replicate the issue. What type of system is your node running on?

Thadah commented 1 year ago

The system is running on OpenMediaVault 6.4.5 (Debian) with the kernel 6.1.0 The CPU is Xeon E5-2680 v4 if that helps

lilkingjr1 commented 1 year ago

So, there's a couple things to note...

Does Arma boot without headless clients? Because all your logs show is that the HC was "launched" (which could still mean it crashed in the background) and that the main Arma server crashed. If Arma crashes without the HC, the issue has nothing to do with HCs, which is what I expect is happening.
Your CPU is now 11 years old, and in the past we've seen issues with CPUs this old having inconsistencies in instruction set versions, causing unexplainable segmentation faults in up-to-date software like Arma.
I'm not an expert at this stuff, but if your distro is Debian or a fork of Debian, all I know is (according to Wikipedia) Debian's latest kernel version is 6.1 on Bookworm (which just came out). I also know Arma's image runs on Bullseye, so maybe there's a kernel discrepancy Arma doesn't like?... Idk... I doubt it, but I know really old kernels in the past don't play well with Arma either.

Thadah commented 1 year ago

Hi, to answer your questions

Yes, the server boots normally without HC, even with mods included. I do not know what causes the crash with the HC seeing that the server itself works, it's a really weird issue.
The CPU is a 5th generation Intel, which is old, yes, but not 11 years old. This processor is 7 years old since its launch and supports the AVX2 instruction set. This processor shouldn't have any problems running Arma as it has all the instructions needed.
I really don't know. I have another server running Ubuntu 22.04 and it isn't having any issues booting the server, although I must say it's using an old egg from before the cleanup you did to it.

lilkingjr1 commented 1 year ago

Hmm... Interesting. Do you mind posting a full boot log with HC logging enabled?
My bad. I accidentally looked up the wrong version of the CPU, and v4 looks like it released Q1'16; not Q1'12.
"I have another server running Ubuntu 22.04 and it isn't having any issues booting the server": Do you mean it doesn't have issues booting with a HC as well? "it's using an old egg from before the cleanup you did to it.": Can you clarify which cleanup you're referring to? If you're referring to my original clean up from Jan 2021, it didn't even have HC support back then and used a Quay image. If you're referring to the last update the egg got, the Yolk image didn't change, so I would be surprised to hear if an older egg version works today with HC vs. the most up to date one. If that's the case, please let me know. Otherwise, I'm not exactly sure what point you're trying to make with part 3 (assuming you're trying to make a point and not just adding more information which is fine).

Thadah commented 1 year ago

This is the whole log Pterodactyl creates with HC logging enabled. I'm assuming it's the HC the one that crashes because the server starts fine without one. But how can that be? Isn't the HC just another instance of armaserver_x64?

[Pterodactyl Daemon]: Updating process configuration files...
[Pterodactyl Daemon]: Ensuring file permissions are set correctly, this could take a few seconds...
container@pterodactyl~ Server marked as starting...
[Pterodactyl Daemon]: Pulling Docker container image, this could take a few minutes to complete...
Pulling from parkervcp/games 
Status: Image is up to date for ghcr.io/parkervcp/games:arma3 
Digest: sha256:4af582b36360135e765d391d244ccf2c809354866832711b1297becdfe13c7a4 
[Pterodactyl Daemon]: Finished pulling Docker container image
[STARTUP]: Starting checks for all updates...
(It is okay to ignore any "SDL", "steamservice", and "thread priority" errors during this process)
[UPDATE]: Checking for game server updates with App ID: 233780...
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/container/Steam/logs/stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1687387651
-- type 'quit' to exit --
Loading Steam API...dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
dlmopen libSDL3.so.0 failed: libSDL3.so.0: cannot open shared object file: No such file or directory
OK
Logging in user 'redacted' to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
Success! App '233780' already up to date.
[UPDATE]: Game server is up to date!
[STARTUP]: Clearing Headless Client profiles cache...
[STARTUP]: Starting 1 Headless Client(s).
[STARTUP]: Headless Client 1 launched.
[STARTUP]: Starting server with the following startup command:
./arma3server_x64 -ip=0.0.0.0 -port=2302 -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="" -serverMod="" "-noLogs"
23:59:54 SteamAPI initialization failed. Steam features won't be accessible!23:59:54 SteamAPI initialization failed. Steam features won't be accessible!
/entrypoint.sh: line 398:    52 Segmentation fault      (core dumped) ${modifiedStartup}
PTDL_CONTAINER_ERR: There was an error while attempting to run the start command.
container@pterodactyl~ Server marked as offline...
[Pterodactyl Daemon]: ---------- Detected server process in a crashed state! ----------
[Pterodactyl Daemon]: Exit code: 1
[Pterodactyl Daemon]: Out of memory: false

Yes, I was just adding the info that it seems to be happening in this server only. I created a new egg and moved the server to it. I changed the startup to the new default one and started it normally with a HC, so it doesn't seem to be an egg issue.

This is the log from the second server:

[Pterodactyl Daemon]: Updating process configuration files...
[Pterodactyl Daemon]: Ensuring file permissions are set correctly, this could take a few seconds...
container@pterodactyl~ Server marked as starting...
[Pterodactyl Daemon]: Pulling Docker container image, this could take a few minutes to complete...
Pulling from parkervcp/games 
Status: Image is up to date for ghcr.io/parkervcp/games:arma3 
Digest: sha256:4af582b36360135e765d391d244ccf2c809354866832711b1297becdfe13c7a4 
[Pterodactyl Daemon]: Finished pulling Docker container image
[STARTUP]: Starting checks for all updates...
(It is okay to ignore any "SDL", "steamservice", and "thread priority" errors during this process)
[UPDATE]: Checking for game server updates with App ID: 233780...
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/container/Steam/logs/stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1687387651
-- type 'quit' to exit --
Loading Steam API...dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
dlmopen libSDL3.so.0 failed: libSDL3.so.0: cannot open shared object file: No such file or directory
OK
Logging in user 'redacted' to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
Success! App '233780' already up to date.
CWorkThreadPool::~CWorkThreadPool: work complete queue not empty, 1 items discarded.
[UPDATE]: Game server is up to date!
[STARTUP]: Clearing Headless Client profiles cache...
[STARTUP]: Starting 1 Headless Client(s).
[STARTUP]: Headless Client 1 launched.
[STARTUP]: Starting server with the following startup command:
./arma3server_x64 -ip=0.0.0.0 -port=2302 -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="" -serverMod="" "-noLogs"
 0:09:47 SteamAPI initialization failed. Steam features won't be accessible!
 0:09:47 SteamAPI initialization failed. Steam features won't be accessible!
 0:09:47 Waiting for './serverprofile/DataCache/cache_lock' to be available
sh: 1: /sbin/ifconfig: not found
 0:09:48 Waiting for './serverprofile/DataCache/cache_lock' to be available
 0:09:49 Waiting for './serverprofile/DataCache/cache_lock' to be available
 0:09:50 Waiting for './serverprofile/DataCache/cache_lock' to be available
 0:09:51 Waiting for './serverprofile/DataCache/cache_lock' to be available
 0:09:52 Waiting for './serverprofile/DataCache/cache_lock' to be available
container@pterodactyl~ Server marked as stopping...

The docker images are identical, so it must be something external to the egg and yolk? I don't get what could be causing this behavior... Could it be the SteamAPI initialization failure? It's the only different thing I see.

lilkingjr1 commented 1 year ago

1.

It's likely the main server is crashing when you have HC enabled; not the HC. If the HC was crashing instantly without printing any logs first (assuming you have logging enabled), you would probably get the segfault right after "Headless Client 1 launched", which is when the HC launches. The HC is launched by just calling the same server binary as main, but with different flags, and it is placed in the background.
The logging setting just determines if it's output is directed to the standard console or to /dev/null -- See https://github.com/parkervcp/yolks/blob/1e97785047f90e9ac30ba10c5928b7535dda0826/games/arma3/entrypoint.sh#L387C198-L387C207)

Then the main server is launched during "Starting server with the following startup command". However, the HC, before the main starts, puts a lock on a shared file that is used to start the server, and then is eventually released for main to then use to boot. Main might be crashing on this locked file for you, but again, it still works for most people and looks like the following (note the cache_lock):

[UPDATE]: Game server is up to date!
[UPDATE]: Checking all Steam Workshop mods for updates...
[UPDATE]: Steam Workshop mod update check complete!
[STARTUP]: Starting 1 Headless Client(s).
[STARTUP]: Headless Client 1 launched.
[STARTUP]: Starting server with the following startup command:
./arma3server_x64 -ip=0.0.0.0 -port=25595 -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="@450814997;" -serverMod="" "-noLogs"
22:43:51 SteamAPI initialization failed. Steam features won't be accessible!
22:43:51 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:52 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:53 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:54 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:55 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:56 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:57 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:58 Waiting for './serverprofile/DataCache/cache_lock' to be available
22:43:59 Waiting for './serverprofile/DataCache/cache_lock' to be available
sh: 1: /sbin/ifconfig: not found
22:44:08 Dedicated host created.

I don't understand... So it works for you now?... What did you change? What's different? It sounds like your second server uses the latest egg and image which is good. Is your second server on a different node? If your first server doesn't work, and the only difference is that it's on a different host/node, that's your issue; some weird edge case with your choice of hardware or (more likely) OS.
- "Could it be the SteamAPI initialization failure? It's the only different thing I see.": Doesn't look different to me. It's in both logs, but the first log doesn't have a new line for some reason (maybe that's because the HC and main printed at the same time, idk). It's also unlikely to be the cause of the issue in the first place.

PS. I would disable "Clear Headless Client profiles cache" while troubleshooting.

Thadah commented 1 year ago

I disabled the "Clear Headless Client profiles cache" but it kept crashing. After clicking Start a couple times I managed to get one executed that supposedly had one Headless Client enabled, but it's false. That Headless Client never gets executed nor it joins the server (as it doesn't really exist). I executed it without hiding HC logs, but it doesn't show anything.

[Pterodactyl Daemon]: Updating process configuration files...
[Pterodactyl Daemon]: Ensuring file permissions are set correctly, this could take a few seconds...
container@pterodactyl~ Server marked as starting...
[Pterodactyl Daemon]: Pulling Docker container image, this could take a few minutes to complete...
Pulling from parkervcp/games 
Digest: sha256:4af582b36360135e765d391d244ccf2c809354866832711b1297becdfe13c7a4 
Status: Image is up to date for ghcr.io/parkervcp/games:arma3 
[Pterodactyl Daemon]: Finished pulling Docker container image
[STARTUP]: Starting 1 Headless Client(s).
[STARTUP]: Headless Client 1 launched.
[STARTUP]: Starting server with the following startup command:
./arma3server_x64 -ip=0.0.0.0 -port=2302 -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="@1355571744;@1673456286;@1779063631;@2018593688;@2174495332;@2867537125;@333310405;@450814997;@463939057;@623475643;@751965892;@753946944;@766491311;@773125288;@773131200;@843425103;@843577117;@843593391;@843632231;@866772689;@884966711;" -serverMod="" "-noLogs"
10:21:36 SteamAPI initialization failed. Steam features won't be accessible!
10:21:36 SteamAPI initialization failed. Steam features won't be accessible!
sh: 1: /sbin/ifconfig: not found
10:22:52 Dedicated host created.
CApplicationManagerPopulateThread took 0 milliseconds to initialize (will have waited on CAppInfoCacheReadFromDiskThread)
CAppInfoCacheReadFromDiskThread took 19 milliseconds to initialize
RecordSteamInterfaceCreation (PID 45): SteamGameServer012 / GameServer
RecordSteamInterfaceCreation (PID 45): SteamUtils009 / Utils
Setting breakpad minidump AppID = 107410
10:23:56 Game Port: 2302, Steam Query Port: 2303
RecordSteamInterfaceCreation (PID 45): SteamGameServer012 / GameServer
RecordSteamInterfaceCreation (PID 45): SteamUtils009 / Utils
RecordSteamInterfaceCreation (PID 45): SteamNetworking005 / Networking
RecordSteamInterfaceCreation (PID 45): SteamGameServerStats001 / GameServerStats
RecordSteamInterfaceCreation (PID 45): STEAMHTTP_INTERFACE_VERSION002 / HTTP
RecordSteamInterfaceCreation (PID 45): STEAMINVENTORY_INTERFACE_V002 / Inventory
RecordSteamInterfaceCreation (PID 45): STEAMUGC_INTERFACE_VERSION010 / UGC
RecordSteamInterfaceCreation (PID 45): STEAMAPPS_INTERFACE_VERSION008 / Apps
Arma 3 Console version 2.12.150301 x64 : port 2302
10:23:57 Host identity created.
container@pterodactyl~ Server marked as running...

imagen

Yes, it does work on the second server using the latest egg. It's in a different node and physical hardware, so it's probably a software fault, although I don't know how that could be possible. Should I try changing the default memory allocator to see if it prevents the core dump? I'm at a loss in what to do from this point on. Yes, I could run the server without an HC, but it won't be a great experience as we plan on using Antistasi and it's a heavy mod on AI calculations.

Thadah commented 1 year ago

Adding more HCs to the startup crashes it, but shows other startup logs that weren't showing before, maybe it helps a bit?

[Pterodactyl Daemon]: Finished pulling Docker container image
[STARTUP]: Starting 3 Headless Client(s).
[STARTUP]: Headless Client 1 launched.
[STARTUP]: Headless Client 2 launched.
[STARTUP]: Headless Client 3 launched.
[STARTUP]: Starting server with the following startup command:
./arma3server_x64 -ip=0.0.0.0 -port=2302 -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="@1355571744;@1673456286;@1779063631;@2018593688;@2174495332;@2867537125;@333310405;@450814997;@463939057;@623475643;@751965892;@753946944;@766491311;@773125288;@773131200;@843425103;@843577117;@843593391;@843632231;@866772689;@884966711;" -serverMod="" "-noLogs"
12:25:28 SteamAPI initialization failed. Steam features won't be accessible!
12:25:28 SteamAPI initialization failed. Steam features won't be accessible!
12:25:28 SteamAPI initialization failed. Steam features won't be accessible!
12:25:28 SteamAPI initialization failed. Steam features won't be accessible!
/entrypoint.sh: line 398:    45 Segmentation fault      (core dumped) ./${SERVER_BINARY} -client -connect=127.0.0.1 -port=${SERVER_PORT} -password="${SERVER_PASSWORD}" -profiles=./serverprofile -bepath=./battleye -mod="${CLIENT_MODS}" ${STARTUP_PARAMS}
/entrypoint.sh: line 398:    46 Segmentation fault      ./${SERVER_BINARY} -client -connect=127.0.0.1 -port=${SERVER_PORT} -password="${SERVER_PASSWORD}" -profiles=./serverprofile -bepath=./battleye -mod="${CLIENT_MODS}" ${STARTUP_PARAMS}
/entrypoint.sh: line 398:    47 Segmentation fault      (core dumped) ${modifiedStartup}

lilkingjr1 commented 1 year ago

If it helps you with your troubleshooting:

Segmentation fault (core dumped) ./${SERVER_BINARY} means the HC crashed
Segmentation fault (core dumped) ${modifiedStartup} means the main server crashed

However, I'm going to close this issue as "not feasible / unable to replicate" for now. I think we have run the gambit of troubleshooting that is within the scope of this repo. While unfortunate, we have successfully determined the issue is host specific on your end and not a wider issue with the Egg or Yolk themselves. My leading theory at the moment is OpenMediaVault being your choice of host OS. Arma has been known to have weird, unexplainable issues on hosts with NAS-focused OSes for some reason, even if they are forks of major distros (like Debian in this case). And to be honest, I don't feel like maintaining this already complicated game, egg, and yolk for distros other than base Ubuntu and Debian, especially when the game server barely runs on Linux as is.

Feel free to keep commenting with any additional finds you may have though.

Thadah commented 1 year ago

Hey, thanks for the feedback with identifying the crashes! I was thinking it could be some folder permission that was preventing /serverprofile/DataCache/cache_lock from creating and causing a hard crash, but I've seen it does create it and crashes right after.

I have been testing different things in the last couple of days. So this was the previous situation:

The server will always work when not using HCs
The server will crash when using HCs, or just load the server without them, falsely claiming they are enabled

This is what I found when testing other things:

When executing the server directly from Debian, without using the Yolk, it will work
When executing HCs directly from Debian, the HCs will also work and connect to the main server

Server startup argument:

./arma3server_x64 -ip=0.0.0.0 -port=2302 -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="@1355571744;@1673456286;@1779063631;@2018593688;@2174495332;@2867537125;@333310405;@450814997;@463939057;@623475643;@751965892;@753946944;@766491311;@773125288;@773131200;@843425103;@843577117;@843593391;@843632231;@866772689;@884966711;" -serverMod="" ""

HC startup argument:

./arma3server_x64 -client -connect=127.0.0.1 -port=2302 -password=redacted -profiles=./serverprofile -bepath=./ -cfg=basic.cfg -config=server.cfg -mod="@1355571744;@1673456286;@1779063631;@2018593688;@2174495332;@2867537125;@333310405;@450814997;@463939057;@623475643;@751965892;@753946944;@766491311;@773125288;@773131200;@843425103;@843577117;@843593391;@843632231;@866772689;@884966711;" -serverMod="" "-noLogs"

I believe the issue is being caused by some libraries inside the Docker container, but I don't really have the knowledge to find out what exactly is causing the problem inside the container. If someone can point out debugging steps to find out which library or configuration from the container is causing the crash, I'll gladly test it out.

For now we can make this workaround to execute the server, although we lose any control through Pterodactyl. I'd like to be able to execute it through the panel in the future.

lilkingjr1 commented 1 year ago

@Thadah Thanks for the update. And to be clear, when I said my leading theory was that Arma didn't like running on certain host OSs, I meant running within Ptero / being containerized; not necessarily running on the bare OS. This aligns with what you found, which is good in terms of consistency at least. Could it mean there's an issue with the Yolk? Possibly... But tracking down run time inconsistencies between host OSs within Docker is unfortunately beyond my pay grade 😅

Thadah commented 1 year ago

Hello!

I don't know if it was the last Arma 3 update or the fa7272fe0f871a25dfea44bdf448e2b1ccc61d554aa8cdbdd2f7758505c508de version of the Docker container, but it fixed the problem, we can start headless clients from the panel now.

Thank you @lilkingjr1 for following the issue and trying to help. It was a weird bug indeed, I'm glad it could be solved.

pelican-eggs / eggs