[Bug]: Ark: Survival Evolved scheduled restart / update

tryallthethings commented 2 years ago

Panel Version

1.10.1

Wings Version

1.7.0

Service

Ark: Survival Evolved

Modified

No, I did not modify the egg

Expected Behavior

I've configured a daily restart (via schedules) of both our ARK servers to pull the latest ARK server and mod versions. This worked fine until ~1 month ago.

Schedule:

chrome_2022-08-02_21-28-44

Actual Behavior

The server still restarts and will sometimes pull updates, but never reaches a "running" state every other day. You can neither connect ingame nor via RCON. The only fix I found so far is to stop the server completely and start it again (via Pterodactyl UI). Just restarting (also via UI) does not work. Is there a difference between restart and stop/start?

Steps To Reproduce

Install ARK server (config is pretty much vanilla except a few mod settings, our mods: https://steamcommunity.com/sharedfiles/filedetails/?id=2825462949)
Schedule daily restart for existing ARK server as per screenshot above
Restart will occur but server will not reach playable / connectable state every other day

I'll update with a console log once the issue occurs again. But it's basically just spamming the rcon connection attempt forever:

dial tcp 127.0.0.1:27021: connect: connection refused
waiting for rcon connection...

Are there any other useful logs I could look into? Or is this just ARK weirdness I have to live with? The restart is at 6 AM, the time I notice the issue is usually in the afternoon (6-9 hours later), so the server had sufficient time to start up again.

Logs below are from the last stop / start. Will also update those once the issue occurs again.

Install logs

https://pteropaste.com/idbpzk/

tryallthethings commented 2 years ago

Today it happened again on both of our ARK servers: Console log:

container@pterodactyl~ Server marked as starting...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
waiting for rcon connection...
[remove]
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused

RCON:

Install log from the scheduled restart: https://pteropaste.com/c2nom4/ (not downloading any mods) Install log from the manual restart: https://pteropaste.com/02s02r (did download mods

Console log after manual restart:

container@pterodactyl~ Server marked as starting...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
[... removed duplicate lines ...]
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
container@pterodactyl~ Server marked as stopping...
container@pterodactyl~ Server marked as offline...
[Pterodactyl Daemon]: Updating process configuration files...
[Pterodactyl Daemon]: Ensuring file permissions are set correctly, this could take a few seconds...
container@pterodactyl~ Server marked as starting...
[Pterodactyl Daemon]: Pulling Docker container image, this could take a few minutes to complete...
Pulling from parkervcp/pterodactyl-images 
Digest: sha256:9127081aaf80e38544b183d96d06cf610d865e1b67606f1173ba5f7737d4f1c9 
Status: Image is up to date for quay.io/parkervcp/pterodactyl-images:debian_source 
[Pterodactyl Daemon]: Finished pulling Docker container image
steam user is not set.
Using anonymous user.
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/container/Steam/logs/stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1658944597
-- type 'quit' to exit --
Loading Steam API...OK
Connecting anonymously to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
Please use force_install_dir before logon!
Success! App '376030' already up to date.
:/home/container$ rmv() { echo -e "stopping server"; rcon -t rcon -a 127.0.0.1:${RCON_PORT} -p ${ARK_ADMIN_PASSWORD} -c saveworld && rcon -a 127.0.0.1:${RCON_PORT} -p ${ARK_ADMIN_PASSWORD} -c DoExit; }; trap rmv 15; cd ShooterGame/Binaries/Linux && ./ShooterGameServer ${SERVER_MAP}?listen?SessionName="${SESSION_NAME}"?ServerPassword=${ARK_PASSWORD}?ServerAdminPassword=${ARK_ADMIN_PASSWORD}?Port=27022?PeerPort=27023?PreventDiseases=true?RCONPort=${RCON_PORT}?QueryPort=${QUERY_PORT}?RCONEnabled=True$( [ "$BATTLE_EYE" == "1" ] || printf %s ' -NoBattlEye' ) -server ${ARGS} -log -servergamelog & until echo "waiting for rcon connection..."; rcon -t rcon -a 127.0.0.1:${RCON_PORT} -p ${ARK_ADMIN_PASSWORD}; do sleep 5; done
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
Shutdown handler: initalize.
Using binned.
4.5.1-0+UE4 7038 3077 404 10
[S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.
Setting breakpad minidump AppID = 346110
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/container/Steam/logs/stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
Steam Console Client (c) Valve Corporation - version 1658944597
-- type 'quit' to exit --
Loading Steam API...OK
Connecting anonymously to Steam Public...waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
OK
Waiting for client config...OK
Waiting for user info...OK
Downloading item 893735676 ...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
Success. Downloaded item 893735676 to "/home/container/Steam/steamapps/workshop/content/346110/893735676" (2365355182 bytes) Downloading item 1163444471 ...
Success. Downloaded item 1163444471 to "/home/container/Steam/steamapps/workshop/content/346110/1163444471" (46616097 bytes) Downloading item 1515475686 ...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
Success. Downloaded item 1515475686 to "/home/container/Steam/steamapps/workshop/content/346110/1515475686" (90789881 bytes) Downloading item 731604991 ...
Success. Downloaded item 731604991 to "/home/container/Steam/steamapps/workshop/content/346110/731604991" (60988746 bytes) Downloading item 1231538641 ...
Success. Downloaded item 1231538641 to "/home/container/Steam/steamapps/workshop/content/346110/1231538641" (184116388 bytes) Downloading item 1814953878 ...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
Success. Downloaded item 1814953878 to "/home/container/Steam/steamapps/workshop/content/346110/1814953878" (688444153 bytes) Downloading item 889745138 ...
Success. Downloaded item 889745138 to "/home/container/Steam/steamapps/workshop/content/346110/889745138" (24637315 bytes) Downloading item 1428596566 ...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
Success. Downloaded item 1428596566 to "/home/container/Steam/steamapps/workshop/content/346110/1428596566" (1894087 bytes) Downloading item 849372965 ...
Success. Downloaded item 849372965 to "/home/container/Steam/steamapps/workshop/content/346110/849372965" (94345231 bytes) Downloading item 1609138312 ...
Success. Downloaded item 1609138312 to "/home/container/Steam/steamapps/workshop/content/346110/1609138312" (42924599 bytes) Downloading item 1404697612 ...
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
Success. Downloaded item 1404697612 to "/home/container/Steam/steamapps/workshop/content/346110/1404697612" (3418407 bytes) Downloading item 1772839535 ...
Success. Downloaded item 1772839535 to "/home/container/Steam/steamapps/workshop/content/346110/1772839535" (351325507 bytes) Downloading item 1591643730 ...
Success. Downloaded item 1591643730 to "/home/container/Steam/steamapps/workshop/content/346110/1591643730" (505437 bytes) waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
[... removed duplicate lines ...]
waiting for rcon connection...
dial tcp 127.0.0.1:27021: connect: connection refused
waiting for rcon connection...
read tcp 127.0.0.1:47546->127.0.0.1:27021: i/o timeout
[... removed duplicate lines ...]
waiting for rcon connection...
read tcp 127.0.0.1:47942->127.0.0.1:27021: i/o timeout
waiting for rcon connection...
dial tcp 127.0.0.1:27021: i/o timeout
[... removed duplicate lines ...]
waiting for rcon connection...
dial tcp 127.0.0.1:27021: i/o timeout
waiting for rcon connection...
read tcp 127.0.0.1:48234->127.0.0.1:27021: i/o timeout
[... removed duplicate lines ...]
read tcp 127.0.0.1:48682->127.0.0.1:27021: i/o timeout
waiting for rcon connection...
dial tcp 127.0.0.1:27021: i/o timeout
[... removed duplicate lines ...]
waiting for rcon connection...
dial tcp 127.0.0.1:27021: i/o timeout
waiting for rcon connection...
Waiting commands for 127.0.0.1:27021 (or type :q to exit)
container@pterodactyl~ Server marked as running...

gOOvER commented 2 years ago

Thats not an egg issue. YOu are using a Modded Version of ARK

tryallthethings commented 2 years ago

This might not be an egg issue, but it only happens with the ARK eggs. Other servers restart just fine via the schedule. And the mods should only come into play AFTER being downloaded - which isn't always happening on the scheduled restart. As per the console log, even the base game isn't being updated. The issue is present again today.

Software-Noob commented 2 years ago

Automated or manual (re)start doesn't change the behavior. Pterodactyl would start a new container each time the server starts with files from the bind volume. Restarting is the same as stopping and starting. That's the beauty of Docker containers; always the same behavior each time on everyone's machine.

I don't see anything in this bug report that would be related to the egg. RCON is only available once the server has started and is required for us to have console output due to the lack of a native console in Ark. Failure for it to start only on restarts is most likely your machine running out of resources when multiple servers are booting up at once.

tryallthethings commented 2 years ago

Automated or manual (re)start doesn't change the behavior. Pterodactyl would start a new container each time the server starts with files from the bind volume. Restarting is the same as stopping and starting.

That's what I thought as well. I just noticed the difference in the behavior, hence asking that question.

Failure for it to start only on restarts is most likely your machine running out of resources when multiple servers are booting up at once.

I highly doubt that. It's a very powerful machine, barely scratching 20% CPU utilization and plenty of RAM available at any time. But I'll try rebooting the ARK servers at different times.

Thanks for your reply!

pelican-eggs / eggs