Server instability and crashes due to higher-than-expected ram usage (possible memory leak)

T-h-o-r-p-e commented 1 month ago

Describe the bug

Server instability due to higher-than-expected ram usage (possible memory leak). After some hours of playing, the ram usage of the server grows higher and higher, until a point that the server crashes and restarts. At which point the ram returns to expected usage values. The issue does not seem to be related to the number of players, what is going on on the server, nor if users are online or not. It appears to simply occur as uptime increases. Restarting the server therefore restarts the 'countdown' to the next crash. (Restarting once a day is not enough)

To Reproduce

Steps to reproduce the behavior:

Launch server
Wait
RAM increases over time
Crash and automatic restart n. if the server is restarted before a crash, the time until the next crash appears to reset

Expected behaviour

RAM usage stays relatively stable and does not climb uncapped to a point of a crash

Actual behavior

Total RAM usage by the server increases over time until it uses the max amount of RAM (and in my case Swap Memory) and then crashes the server.

Screenshots

OS information (IMPORTANT)

OS: Ubuntu
Palworld Version: 0.3.2.55496
Docker Image Version: 0.38.1

Hardware information (IMPORTANT)

CPU: 4 AMD VCPU on Hetzner Cloud Server
Memory: 8GB plus 16GB Swap

docker-compose.yml contents (IMPORTANT)

services:
  palworld:
    image: thijsvanloef/palworld-server-docker:latest
    restart: unless-stopped
    container_name: palworld-server
    stop_grace_period: 30s  # Set to however long you are willing to wait for the container to gracefully stop
    ports:
      - 8211:8211/udp
      - 27015:27015/udp  # Required if you want your server to show up in the community servers tab
    environment:
      PUID: 1000
      PGID: 1000
      PORT: ### # Optional but recommended
      PLAYERS: 32  # Optional but recommended
      SERVER_PASSWORD: "###"  # Optional but recommended
      MULTITHREADING: "true"
      RCON_ENABLED: "true"
      RCON_PORT: ###
      TZ: "UTC"
      ADMIN_PASSWORD: "###"
      COMMUNITY: "false"  # Enable this if you want your server to show up in the community servers tab, USE WITH SERVER_PASSWORD!
      SERVER_NAME: "###"
      SERVER_DESCRIPTION: "Work God Dammit"
      EXP_RATE: 2
      PAL_CAPTURE_RATE: 1.5
      PAL_SPAWN_NUM_RATE: 1.9
      PLAYER_AUTO_HP_REGEN_RATE: 1.5
      PLAYER_AUTO_HP_REGEN_RATE_IN_SLEEP: 5
      PAL_AUTO_HP_REGEN_RATE: 2
      PAL_AUTO_HP_REGEN_RATE_IN_SLEEP: 5
      BUILD_OBJECT_DAMAGE_RATE: 0.25
      BUILD_OBJECT_DETERIORATION_DAMAGE_RATE: 0
      COLLECTION_DROP_RATE: 2
      COLLECTION_OBJECT_RESPAWN_SPEED_RATE: 0.5
      ENEMY_DROP_ITEM_RATE: 2
      DEATH_PENALTY: 1
      BASE_CAMP_WORKER_MAX_NUM: 25
      GUILD_PLAYER_MAX_NUM: 32
      PAL_EGG_DEFAULT_HATCHING_TIME: 0.5
      WORK_SPEED_RATE: 3
      ENABLE_NON_LOGIN_PENALTY: "false"
      COOP_PLAYER_MAX_NUM: 32
      DISCORD_WEBHOOK_URL: ###
      DISCORD_PRE_START_MESSAGE: "Server has started!"
      DISCORD_PRE_START_MESSAGE_ENABLED: "true"
      DISCORD_POST_SHUTDOWN_MESSAGE: "Server has stopped!"
      DISCORD_POST_SHUTDOWN_MESSAGE_ENABLED: "true"
      DISCORD_PLAYER_JOIN_MESSAGE_ENABLED: "true"
      DISCORD_PLAYER_LEAVE_MESSAGE_ENABLED: "true"
      DISCORD_PRE_BACKUP_MESSAGE_ENABLED: "false"
      DISCORD_POST_BACKUP_MESSAGE_ENABLED: "false"
      AUTO_REBOOT_ENABLED: "true"
      AUTO_REBOOT_WARN_MINUTES: 1
      BASE_CAMP_MAX_NUM_IN_GUILD: 10
      AUTO_SAVE_SPAN: 15
      INVISIBLE_OTHER_GUILD_BASE_CAMP_AREA_FX: "false"

    volumes:
      - ./palworld:/palworld/

Container/Host Logs (IMPORTANT)

Not sure where to locate this

Additional context

Bug Report Checklist

[Y ] I searched for my issue on the Known issues page
[Y ] I searched for my issue in the Closed issues
[Y ] I have included all relevant information required above.
[- ] I am willing to fix this bug

MamboDead commented 1 month ago

Facing the exact same issue here on my Docker Server. Seems related to the Sakura update.

Before the update the server usually stayed around 6 GB RAM Usage, after the update it takes whatever is available until OS kills the process for memory reasons.

Grafana Memory overview of last night

18h is daily server restart. Around 19:30 did a manual restart for config changes. 02:45 is when the server crashed cause memory usage. and finally around 04:15 everyone stopped playing.

Checking my Grafana logs over a period of 7 Days indicates that issue started since 27/06 (Sakura Update day)

If anything I can provide to assist let me know.

thijsvanloef commented 1 month ago

Hi @MamboDead & @T-h-o-r-p-e,

Sadly I have no control over the memory usage of the application that is running in the container. I am afraid that this needs to be handled by the Palworld Devs.

thijsvanloef / palworld-server-docker