[Feature Request] Is there a way to safe restart the server via cron or something? (Memory Leak)

dakeeper commented 5 months ago

Have you read the Important information text above

[X] Yes i did

Describe the feature

Hello,

is there a feature to restart the server at a given time? This will help till the memory leak is fixed. Maybe with a ingame announcement before restart?

any suggestions?

Thank you

Additional information

[ ] Would you be willing to help implement this feature?

Final checks

[X] I understand that this is a feature request function. I also unterstand that if i use this to report a bug report, that my issue might be un-answered closed.

takato1314 commented 5 months ago

i havent dive into all the codes in this repository since i just joined but it does sound like a cronjob will be able to resolve this issue if we set timer to reset at a specific time daily?

jammsen commented 5 months ago

Hey @dakeeper im working on a experiment on this to solve this in a way. Here are some problems to consider:

I, personally dont know, if you RAM is on a VM, Windows-Gaming-System or a QNAP NAS as a few examples
I dont know if you only run docker or other native linux services
I dont know if you are running multiple gameservers or other services in docker-containers too

As you can see its not that easy to say, oh well 80% RAM usage, kill that palworld-ds and restart it. 80% might just what you aim for because you use your docker-node a lot and 80% is a safe-space because you have like 64-128GB RAM. While this is different on a QNAS NAS with "maybe" 16GB RAM?

Its not that cut and dry, thats why im still in experiment phase about this one, but i yesterday already added native rcon/rconcli to the container to enable the change in the future, so the container can do rcon save before restarting 😄

Also i had to establish a few functions first to make this even possible.

TL;DR: Im working on a good solution, please be patient.

dakeeper commented 5 months ago

Find ich gut :)

Fabyloso commented 5 months ago

Hey @dakeeper im working on a experiment on this to solve this in a way. Here are some problems to consider:
* I, personally dont know, if you RAM is on a VM, Windows-Gaming-System or a QNAP NAS as a few examples

* I dont know if you only run docker or other native linux services

* I dont know if you are running multiple gameservers or other services in docker-containers too
As you can see its not that easy to say, oh well 80% RAM usage, kill that palworld-ds and restart it. 80% might just what you aim for because you use your docker-node a lot and 80% is a safe-space because you have like 64-128GB RAM. While this is different on a QNAS NAS with "maybe" 16GB RAM?

Its not that cut and dry, thats why im still in experiment phase about this one, but i yesterday already added native rcon/rconcli to the container to enable the change in the future, so the container can do rcon save before restarting 😄

Also i had to establish a few functions first to make this even possible.

TL;DR: Im working on a good solution, please be patient.

You might be over thinking by keying off of the RAM usage, a cron job that the user can set would be the easiest thing to implement. I only have about 4 players on my server and it will crash within 48hrs. Like you said it's hard to figure out everyone's config, just let the users decide how long the server should be up.

At the moment I just run a cron job on my machine to just restart this container at 4am, and it probably take a minute to restart. If anyone's up and playing at that time its not a huge inconvenience.

jammsen commented 5 months ago

Hey @dakeeper im working on a experiment on this to solve this in a way. Here are some problems to consider:
* I, personally dont know, if you RAM is on a VM, Windows-Gaming-System or a QNAP NAS as a few examples

* I dont know if you only run docker or other native linux services

* I dont know if you are running multiple gameservers or other services in docker-containers too
As you can see its not that easy to say, oh well 80% RAM usage, kill that palworld-ds and restart it. 80% might just what you aim for because you use your docker-node a lot and 80% is a safe-space because you have like 64-128GB RAM. While this is different on a QNAS NAS with "maybe" 16GB RAM? Its not that cut and dry, thats why im still in experiment phase about this one, but i yesterday already added native rcon/rconcli to the container to enable the change in the future, so the container can do rcon save before restarting 😄 Also i had to establish a few functions first to make this even possible. TL;DR: Im working on a good solution, please be patient.
You might be over thinking by keying off of the RAM usage, a cron job that the user can set would be the easiest thing to implement. I only have about 4 players on my server and it will crash within 48hrs. Like you said it's hard to figure out everyone's config, just let the users decide how long the server should be up.

At the moment I just run a cron job on my machine to just restart this container at 4am, and it probably take a minute to restart. If anyone's up and playing at that time its not a huge inconvenience.

So you are saying you are managing this outside of the container not inside and users should rather go that route?

Fabyloso commented 5 months ago

So you are saying you are managing this outside of the container not inside and users should rather go that route?

No I think this is a good feature. I'm just saying ignore the RAM usage route and just setup a cron job for this container to restart at some user designated time. The Palworld devs will eventually fix the mem leak problem and this feature would end up being unnecessary, so just take the easiest path.

Any power or determined enough user could setup a cron job but I'm assuming this would be good for the average user. Assuming the average user only follows guides to set these things up and doesn't understand what is actually going on.

jammsen commented 5 months ago

So you are saying you are managing this outside of the container not inside and users should rather go that route?

No I think this is a good feature. I'm just saying ignore the RAM usage route and just setup a cron job for this container to restart at some user designated time. The Palworld devs will eventually fix the mem leak problem and this feature would end up being unnecessary, so just take the easiest path.

Any power or determined enough user could setup a cron job but I'm assuming this would be good for the average user. Assuming the average user only follows guides to set these things up and doesn't understand what is actually going on.

So a cron entry to just stop the server to a defned interval and let docker-compose deal with the rest. Hmmmm yeah that could be a good solution.

mzettwitz commented 5 months ago

I was hoping you guys are working on a fix since I googled for a solution during my lunch break and came across a similar solution for docker: https://www.number13.de/palworld-restart-script-fuer-server-einrichten/

I am not very familiar with cron jobs, but is it possible to define multiple fixed day times? Like 3 a.m. and 4 p.m. since a coworker said that it is recommended to restart the server every like every 8 hours? An ingame announcement using rcon would be sweet, too.

Btw. thanks for your great work and passion in this project. I can imagine you would like to play more the game than care about the server setup 😄

Fabyloso commented 5 months ago

@mzettwitz it depends on your system, the more RAM you have the longer it will go without crashing. Use this link to get whatever time you want setup, run at 4a every other day and then your command will look something like, 0 4 */2 * * docker compose -f palworld-dedicated-server restart - every other day at 4a 0 4 * * * docker compose -f palworld-dedicated-server restart - every day at 4a assuming palworld-dedicated-server is the name of your container.

I wouldn't bother with anything more than that. The devs will fix this issue soon, and you don't really need to restart that often.

Compupaq commented 5 months ago

Here's a bash script I run as a cron job for my server, outside of docker. Maybe it'll be useful for or inspire others. I'm not sure if force-saving and shutting down the Palworld server application makes any difference or not, but it shouldn't hurt.

#!/bin/bash

# Broadcast a restart warning message
rcon -a localhost:25575 -p <password> "Broadcast Server_will_be_restarting_in_5_minutes."

# Wait 4 minutes before continuing
sleep 240

# Shut down the server with a 60 second timer
rcon -a localhost:25575 -p <password> "Shutdown 60"

# Wait a bit, then save
sleep 30
rcon -a localhost:25575 -p <password> "Save"

# Wait for the server to be fully shut down
sleep 60

# Stop the container, update it, and start/recreate it
cd /path/to/docker-compose/file/
docker compose stop
docker compose pull
docker compose up -d

CooperWallace commented 5 months ago

I run the following in an hourly cronjob which will auto restart when a certain threshold is reached.

#!/bin/bash
IP='127.0.0.1:25575'
PASS=''
THRESHOLD=75

cd <palworld-path>

MEMORY_USAGE=$(free | grep Mem | awk '{print $3/$2 * 100.0}')
echo "$(date): Memory usage is at $MEMORY_USAGE%.." >> mem.log

if (( $(echo "$MEMORY_USAGE > $THRESHOLD" | bc -l) )); then
    echo "$(date) Restarting. Running clean command." >> mem.log
    echo "broadcast Restarting_due_to_memory_threshold_reached."  | rcon -a $IP -p $PASS
    ./restart.sh
fi

Can be adapted as needed. Was quick and dirty but someone might find it useful

jammsen commented 5 months ago

I wouldn't bother with anything more than that. The devs will fix this issue soon, and you don't really need to restart that often.

I know what you mean, but i wouldnt bank on it, beeing soon to fix all the problems. The server side mostly doesnt see the light of day, while clients can crash and give a bad-exp. Thats where the focus will be, is my assumtion.

since a coworker said that it is recommended to restart the server every like every 8 hours? @mzettwitz it depends on your system, the more RAM you have the longer it will go without crashing. Use this link to get whatever time you want setup, run at 4a every other day and then your command will look something like, 0 4 */2 * * docker compose -f palworld-dedicated-server restart - every other day at 4a 0 4 * * * docker compose -f palworld-dedicated-server restart - every day at 4a assuming palworld-dedicated-server is the name of your container.

Well 8 hours would be this here: https://cron.help/#0_*/8_*_*_* - But he is right, depends and your setup and progress. You will have to find a balance yourself 😄

Here's a bash script I run as a cron job for my server, outside of docker. Maybe it'll be useful for or inspire others. I'm not sure if force-saving and shutting down the Palworld server application makes any difference or not, but it shouldn't hurt.
#!/bin/bash

# Broadcast a restart warning message
rcon -a localhost:25575 -p <password> "Broadcast Server_will_be_restarting_in_5_minutes."

# Wait 4 minutes before continuing
sleep 240

# Shut down the server with a 60 second timer
rcon -a localhost:25575 -p <password> "Shutdown 60"

# Wait a bit, then save
sleep 30
rcon -a localhost:25575 -p <password> "Save"

# Wait for the server to be fully shut down
sleep 60

# Stop the container, update it, and start/recreate it
cd /path/to/docker-compose/file/
docker compose stop
docker compose pull
docker compose up -d

From experience of corrupted savegames, its saver to call Save first and then Shutdown second. That way its only a matter of time before something weird happens.

I run the following in an hourly cronjob which will auto restart when a certain threshold is reached.

#!/bin/bash
IP='127.0.0.1:25575'
PASS=''
THRESHOLD=75

cd <palworld-path>

MEMORY_USAGE=$(free | grep Mem | awk '{print $3/$2 * 100.0}')
echo "$(date): Memory usage is at $MEMORY_USAGE%.." >> mem.log

if (( $(echo "$MEMORY_USAGE > $THRESHOLD" | bc -l) )); then
    echo "$(date) Restarting. Running clean command." >> mem.log
    echo "broadcast Restarting_due_to_memory_threshold_reached."  | rcon -a $IP -p $PASS
    ./restart.sh
fi

Can be adapted as needed. Was quick and dirty but someone might find it useful

This only works if you only do Palworld in your docker-node, but not if you have other stuff running. Also 75% is different on 16, 32, 64 and 128GB RAM, thats what i tried to say.

jammsen commented 5 months ago

I was hoping you guys are working on a fix since I googled for a solution during my lunch break and came across a similar solution for docker: https://www.number13.de/palworld-restart-script-fuer-server-einrichten/

I am not very familiar with cron jobs, but is it possible to define multiple fixed day times? Like 3 a.m. and 4 p.m. since a coworker said that it is recommended to restart the server every like every 8 hours? An ingame announcement using rcon would be sweet, too.

Btw. thanks for your great work and passion in this project. I can imagine you would like to play more the game than care about the server setup 😄

Also yes, @thejcpalma is working on something for this.

thejcpalma commented 5 months ago

Yes! If PR gets accepted, the bare-bones of the implementation are there, and I'll push the new functionalities right after in a new PR 😉

CooperWallace commented 5 months ago

This only works if you only do Palworld in your docker-node, but not if you have other stuff running. Also 75% is different on 16, 32, 64 and 128GB RAM, thats what i tried to say.

Well of course thats why its a tunable parameter. Theres definitely no one size fits all solution to this so I think its better to let the user dictate it based on their system. Thats also ran outside the container so probably not applicable. Threw it here in case someone else needs it

nickmartin1ee7 commented 5 months ago

I run the following in an hourly cronjob which will auto restart when a certain threshold is reached.

#!/bin/bash
IP='127.0.0.1:25575'
PASS=''
THRESHOLD=75

cd <palworld-path>

MEMORY_USAGE=$(free | grep Mem | awk '{print $3/$2 * 100.0}')
echo "$(date): Memory usage is at $MEMORY_USAGE%.." >> mem.log

if (( $(echo "$MEMORY_USAGE > $THRESHOLD" | bc -l) )); then
    echo "$(date) Restarting. Running clean command." >> mem.log
    echo "broadcast Restarting_due_to_memory_threshold_reached."  | rcon -a $IP -p $PASS
    ./restart.sh
fi

Can be adapted as needed. Was quick and dirty but someone might find it useful

I run this, but closer to every minute. I also check if all my SWAP memory is used as well beforehand.

calvinli723 commented 5 months ago

I think being able to set the cron interval in a similar variable as the Backup cron would be helpful. The main benefit of having a built-in cron instead of adding your own cron outside of the container to restart it, is that the internal cron can maybe give an in-game warning/countdown until the restart happens so players still in the server can get a heads up.

jammsen commented 5 months ago

I think being able to set the cron interval in a similar variable as the Backup cron would be helpful. The main benefit of having a built-in cron instead of adding your own cron outside of the container to restart it, is that the internal cron can maybe give an in-game warning/countdown until the restart happens so players still in the server can get a heads up.

That would be also my most favorite way to do this.

jammsen commented 5 months ago

This is fix in the release on latest from 5 minutes ago. See Changelogs or Discord message regarding the new release and what to lookout for.

Im closing this issue as resolved, feel free to reach out again if you need help.

If you like this project, please consider giving this repo and the Docker-Hub-Repo a Star.

jammsen / docker-palworld-dedicated-server