memory leak in build 391 (commit 3e73355)

[waseric]

Timings or Profile link

https://timings.aikar.co/?id=720d1dde377944f0b59b8a86fd5c51e7

Description of issue

After switch to build 389, memory leak started to occur in one server. server has 4G allocated. Over course of about 4 hours, grows to over 40G mem used. Server eventually fails. Stop, restart, behavior repeats. Switched back to 388 at this point (based on nature of commit 3e73355 and heap dump comparison highlighting leakage in LWCX plugin). Issue no longer occurs.

Plugin and Datapack List

[00:55:11 INFO]: Plugins (36): AncientGates, AreaShop, BanManager, BossShopPro, BungeeTabListPlus, ChatControl, CheckNameHistory, CMILib, CoreProtect, DiscordSRV, dynmap, Essentials, EssentialsSpawn, EventLogger, FastAsyncWorldEdit (WorldEdit), floodgate, GriefPrevention, HubKick, Jobs, LeaderHeadsRevamped, LuckPerms, LWC, Multiverse-Core, NashornPlus, OreAnnouncer, PlaceholderAPI, Plan, PremiumVanish, ProtocolLib, ServerSigns, Skript, sthorses, Vault, ViaVersion, WorldBorder, WorldGuard

[00:55:36 INFO]: There are 5 data packs enabled: [vanilla (built-in)], [file/bukkit (world)], [file/extra-mob-heads-1.17 (world)], [file/dragon drops-1.17 (world)], [file/double shulker shells-1.17 (world)] [00:55:36 INFO]: There are 3 data packs available: [file/extra-mob-heads (world)], [file/extra-mob-heads-1.15 (world)], [file/more-mob-heads-2.3.0 (world)]

Server config files

Paste configs or paste.gg link here!

# This is the main configuration file for Bukkit.
# As you can see, there's actually not that much to configure without any plugins.
# For a reference for any variable inside this file, check out the Bukkit Wiki at
# https://www.spigotmc.org/go/bukkit-yml
# 
# If you need help on this file, feel free to join us on irc or leave a message
# on the forums asking for advice.
# 
# IRC: #spigot @ irc.spi.gt
#    (If this means nothing to you, just go to https://www.spigotmc.org/go/irc )
# Forums: https://www.spigotmc.org/
# Bug tracker: https://www.spigotmc.org/go/bugs

settings:
  allow-end: true
  warn-on-overload: true
  permissions-file: permissions.yml
  update-folder: update
  plugin-profiling: false
  connection-throttle: 4000
  query-plugins: true
  deprecated-verbose: default
  shutdown-message: Server closed
  minimum-api: none
spawn-limits:
  water-underground-creature: 5
  water-ambient: 20
  monsters: 24
  animals: 9
  water-animals: 24
  ambient: 5
chunk-gc:
  period-in-ticks: 400
ticks-per:
  water-underground-creature-spawns: 1
  water-ambient-spawns: 1
  water-spawns: 1
  ambient-spawns: 1
  animal-spawns: 400
  monster-spawns: 4
  autosave: 6000
aliases: now-in-commands.yml
```bukkit.yml

```spigot.yml
# This is the main configuration file for Spigot.
# As you can see, there's tons to configure. Some options may impact gameplay, so use
# with caution, and make sure you know what each option does before configuring.
# For a reference for any variable inside this file, check out the Spigot wiki at
# http://www.spigotmc.org/wiki/spigot-configuration/
# 
# If you need help with the configuration or have any questions related to Spigot,
# join us at the IRC or drop by our forums and leave a post.
# 
# IRC: #spigot @ irc.spi.gt ( http://www.spigotmc.org/pages/irc/ )
# Forums: http://www.spigotmc.org/

config-version: 12
settings:
  log-named-deaths: true
  log-villager-deaths: true
  save-user-cache-on-stop-only: false
  sample-count: 12
  bungeecord: true
  player-shuffle: 0
  user-cache-size: 1000
  moved-wrongly-threshold: 0.0625
  moved-too-quickly-multiplier: 10.0
  timeout-time: 60
  restart-on-crash: true
  restart-script: ./easy.sh
  netty-threads: 4
  debug: false
  attribute:
    maxHealth:
      max: 2048.0
    movementSpeed:
      max: 2048.0
    attackDamage:
      max: 2048.0
messages:
  whitelist: You are not whitelisted on this server!
  unknown-command: Unknown command. Type "/help" for help.
  server-full: The server is full!
  outdated-client: Outdated client! Please use {0}
  outdated-server: Outdated server! I'm still on {0}
  restart: Server is restarting
commands:
  spam-exclusions:
  - /skill
  silent-commandblock-console: false
  replace-commands:
  - setblock
  - summon
  - testforblock
  - tellraw
  log: true
  tab-complete: 0
  send-namespaced: true
advancements:
  disable-saving: false
  disabled:
  - minecraft:story/disabled
stats:
  disable-saving: false
  forced-stats: {}
world-settings:
  default:
    thunder-chance: 100000
    seed-bastion: 30084232
    seed-fortress: 30084232
    seed-mansion: 10387319
    seed-fossil: 14357921
    seed-portal: 34222645
    end-portal-sound-radius: 0
    seed-endcity: 10387313
    trident-despawn-rate: 1200
    verbose: true
    mob-spawn-range: 4
    hopper-amount: 1
    dragon-death-sound-radius: 0
    seed-village: 10387312
    seed-desert: 14357617
    seed-igloo: 14357618
    seed-jungle: 14357619
    seed-swamp: 14357620
    seed-monument: 10387313
    seed-shipwreck: 165745295
    seed-ocean: 14357621
    seed-outpost: 165745296
    seed-slime: 987234911
    max-tnt-per-tick: 100
    view-distance: default
    arrow-despawn-rate: 600
    enable-zombie-pigmen-portal-spawns: true
    item-despawn-rate: 6000
    wither-spawn-sound-radius: 0
    hanging-tick-frequency: 100
    zombie-aggressive-towards-villager: true
    nerf-spawner-mobs: false
    max-entity-collisions: 8
    growth:
      cactus-modifier: 100
      cane-modifier: 100
      melon-modifier: 100
      mushroom-modifier: 100
      pumpkin-modifier: 100
      sapling-modifier: 100
      beetroot-modifier: 100
      carrot-modifier: 100
      potato-modifier: 100
      wheat-modifier: 100
      netherwart-modifier: 100
      vine-modifier: 100
      cocoa-modifier: 100
      bamboo-modifier: 100
      sweetberry-modifier: 100
      kelp-modifier: 100
    entity-activation-range:
      villagers: 32
      flying-monsters: 32
      villagers-work-immunity-after: 100
      villagers-work-immunity-for: 20
      villagers-active-for-panic: true
      water: 8
      animals: 16
      monsters: 24
      raiders: 48
      misc: 8
      tick-inactive-villagers: false
      wake-up-inactive:
        animals-max-per-tick: 4
        animals-every: 1200
        animals-for: 100
        monsters-max-per-tick: 8
        monsters-every: 400
        monsters-for: 100
        villagers-max-per-tick: 4
        villagers-every: 600
        villagers-for: 100
        flying-monsters-max-per-tick: 8
        flying-monsters-every: 200
        flying-monsters-for: 100
    entity-tracking-range:
      players: 48
      animals: 48
      monsters: 48
      misc: 32
      other: 64
    ticks-per:
      hopper-transfer: 8
      hopper-check: 1
    hunger:
      jump-walk-exhaustion: 0.05
      jump-sprint-exhaustion: 0.2
      combat-exhaustion: 0.1
      regen-exhaustion: 6.0
      swim-multiplier: 0.01
      sprint-multiplier: 0.1
      other-multiplier: 0.0
    max-tick-time:
      tile: 50
      entity: 50
    squid-spawn-range:
      min: 45.0
    merge-radius:
      exp: 3.0
      item: 2.5
  worldeditregentempworld:
    verbose: false
players:
  disable-saving: false

```spigot.yml

```paper.yml
# This is the main configuration file for Paper.
# As you can see, there's tons to configure. Some options may impact gameplay, so use
# with caution, and make sure you know what each option does before configuring.
# 
# If you need help with the configuration or have any questions related to Paper,
# join us in our Discord or IRC channel.
# 
# Discord: https://discord.gg/papermc
# IRC: #paper @ irc.esper.net ( https://webchat.esper.net/?channels=paper ) 
# Website: https://papermc.io/ 
# Docs: https://paper.readthedocs.org/ 

use-display-name-in-quit-message: false
verbose: false
config-version: 24
settings:
  log-player-ip-addresses: true
  lag-compensate-block-breaking: true
  send-full-pos-for-hard-colliding-entities: true
  fix-entity-position-desync: true
  track-plugin-scoreboards: false
  max-joins-per-tick: 3
  player-auto-save-rate: -1
  max-player-auto-save-per-tick: -1
  console-has-all-permissions: false
  chunk-tasks-per-tick: 1000
  load-permissions-yml-before-plugins: true
  bungee-online-mode: true
  incoming-packet-spam-threshold: 300
  use-alternative-luck-formula: false
  use-versioned-world: false
  suggest-player-names-when-null-tab-completions: true
  save-player-data: true
  region-file-cache-size: 256
  enable-player-collisions: true
  save-empty-scoreboard-teams: false
  velocity-support:
    enabled: false
    online-mode: false
    secret: ''
  async-chunks:
    enable: true
    threads: -1
  watchdog:
    early-warning-every: 5000
    early-warning-delay: 10000
  spam-limiter:
    recipe-spam-increment: 1
    recipe-spam-limit: 20
    tab-spam-increment: 1
    tab-spam-limit: 500
  book-size:
    page-max: 2560
    total-multiplier: 0.98
  unsupported-settings:
    allow-headless-pistons: false
    allow-permanent-block-break-exploits: false
    allow-piston-duplication: false
    allow-headless-pistons-readme: This setting controls if players should be able
      to create headless pistons.
    allow-permanent-block-break-exploits-readme: This setting controls if players
      should be able to break bedrock, end portals and other intended to be permanent
      blocks.
    allow-piston-duplication-readme: This setting controls if player should be able
      to use TNT duplication, but this also allows duplicating carpet, rails and potentially
      other items
  console:
    enable-brigadier-highlighting: true
    enable-brigadier-completions: true
  item-validation:
    display-name: 8192
    loc-name: 8192
    lore-line: 8192
    book:
      title: 8192
      author: 8192
      page: 16384
  use-display-name-in-quit-message: false
  loggers:
    deobfuscate-stacktraces: true
  chunk-loading:
    min-load-radius: 2
    max-concurrent-sends: 2
    autoconfig-send-distance: true
    target-player-chunk-send-rate: 100.0
    global-max-chunk-send-rate: -1.0
    enable-frustum-priority: false
    global-max-chunk-load-rate: -1.0
    player-max-concurrent-loads: 4.0
    global-max-concurrent-loads: 500.0
  packet-limiter:
    kick-message: '&cSent too many packets'
    limits:
      all:
        interval: 7.0
        max-packet-rate: 500.0
      PacketPlayInAutoRecipe:
        interval: 4.0
        max-packet-rate: 5.0
        action: DROP
messages:
  no-permission: '&cI''m sorry, but you do not have permission to perform this command.
    Please contact the server administrators if you believe that this is in error.'
  kick:
    authentication-servers-down: ''
    connection-throttle: Connection throttled! Please wait before reconnecting.
    flying-player: Flying is not enabled on this server
    flying-vehicle: Flying is not enabled on this server
timings:
  url: https://timings.aikar.co/
  enabled: true
  verbose: true
  server-name-privacy: false
  hidden-config-entries:
  - database
  - settings.bungeecord-addresses
  - settings.velocity-support.secret
  history-interval: 300
  history-length: 3600
  server-name: Unknown Server
world-settings:
  default:
    split-overstacked-loot: true
    piglins-guard-chests: true
    map-item-frame-cursor-update-interval: 10
    allow-player-cramming-damage: false
    show-sign-click-command-failure-msgs-to-player: false
    seed-based-feature-search-loads-chunks: false
    fix-items-merging-through-walls: false
    map-item-frame-cursor-limit: 128
    allow-using-signs-inside-spawn-protection: false
    fix-wither-targeting-bug: false
    ender-dragons-death-always-places-dragon-egg: false
    update-pathfinding-on-block-update: true
    only-players-collide: false
    allow-vehicle-collisions: true
    portal-search-vanilla-dimension-scaling: true
    max-leash-distance: 10.0
    should-remove-dragon: false
    fix-climbing-bypassing-cramming-rule: false
    zombies-target-turtle-eggs: true
    delay-chunk-unloads-by: 10s
    phantoms-do-not-spawn-on-creative-players: true
    phantoms-only-attack-insomniacs: true
    light-queue-size: 20
    tracker-update-distance: 1.0
    iron-golems-can-spawn-in-air: false
    zombie-villager-infection-chance: 100.0
    seed-based-feature-search: true
    portal-create-radius: 16
    entities-target-with-follow-range: false
    remove-corrupt-tile-entities: false
    fixed-chunk-inhabited-time: -1
    disable-teleportation-suffocation-check: false
    optimize-explosions: true
    per-player-mob-spawns: true
    portal-search-radius: 128
    filter-nbt-data-from-spawn-eggs-and-related: true
    max-entity-collisions: 8
    disable-creeper-lingering-effect: false
    duplicate-uuid-resolver: saferegen
    duplicate-uuid-saferegen-delete-range: 32
    experience-merge-max-value: -1
    use-vanilla-world-scoreboard-name-coloring: false
    enable-treasure-maps: true
    treasure-maps-return-already-discovered: false
    prevent-moving-into-unloaded-chunks: true
    max-auto-save-chunks-per-tick: 6
    falling-block-height-nerf: 0
    tnt-entity-height-nerf: 0
    disable-thunder: false
    skeleton-horse-thunder-spawn-chance: 0.01
    disable-ice-and-snow: false
    count-all-mobs-for-spawning: false
    keep-spawn-loaded-range: 8
    keep-spawn-loaded: true
    auto-save-interval: -1
    armor-stands-do-collision-entity-lookups: true
    nether-ceiling-void-damage-height: 0
    water-over-lava-flow-speed: 5
    grass-spread-tick-rate: 4
    bed-search-radius: 1
    use-faster-eigencraft-redstone: true
    fix-zero-tick-instant-grow-farms: true
    allow-non-player-entities-on-scoreboards: false
    disable-explosion-knockback: false
    container-update-tick-rate: 1
    parrots-are-unaffected-by-player-movement: false
    prevent-tnt-from-moving-in-water: false
    non-player-arrow-despawn-rate: 20
    creative-arrow-despawn-rate: 20
    armor-stands-tick: false
    spawner-nerfed-mobs-should-jump: false
    allow-leashing-undead-horse: false
    baby-zombie-movement-speed: 0.5
    all-chunks-are-slime-chunks: false
    mob-spawner-tick-rate: 2
    lightning-strike-distance-limit:
      sound: -1
      impact-sound: -1
      flash: -1
    frosted-ice:
      enabled: true
      delay:
        min: 20
        max: 40
    lootables:
      auto-replenish: false
      restrict-player-reloot: true
      reset-seed-on-fill: true
      max-refills: -1
      refresh-min: 12h
      refresh-max: 2d
    hopper:
      ignore-occluding-blocks: true
      cooldown-when-full: true
      disable-move-event: false
    alt-item-despawn-rate:
      enabled: false
      items:
        COBBLESTONE: 300
    game-mechanics:
      disable-mob-spawner-spawn-egg-transformation: true
      fix-curing-zombie-villager-discount-exploit: false
      nerf-pigmen-from-nether-portals: false
      disable-pillager-patrols: false
      scan-for-legacy-ender-dragon: true
      disable-relative-projectile-velocity: false
      disable-chest-cat-detection: false
      shield-blocking-delay: 5
      disable-end-credits: false
      disable-player-crits: false
      disable-sprint-interruption-on-attack: false
      disable-unloaded-chunk-enderpearl-exploit: true
      pillager-patrols:
        spawn-chance: 0.2
        spawn-delay:
          per-player: false
          ticks: 12000
        start:
          per-player: false
          day: 5
    max-growth-height:
      cactus: 3
      reeds: 3
      bamboo:
        max: 16
        min: 11
    fishing-time-range:
      MinimumTicks: 100
      MaximumTicks: 600
    despawn-ranges:
      monster:
        soft: 32
        hard: 128
      creature:
        soft: 32
        hard: 128
      ambient:
        soft: 32
        hard: 128
      underground_water_creature:
        soft: 32
        hard: 128
      water_creature:
        soft: 32
        hard: 128
      water_ambient:
        soft: 32
        hard: 64
      misc:
        soft: 32
        hard: 128
    anti-xray:
      max-block-height: 64
      use-permission: false
      lava-obscures: false
      enabled: false
      engine-mode: 1
      chunk-edge-mode: 2
      max-chunk-section-index: 3
      update-radius: 2
      hidden-blocks:
      - gold_ore
      - iron_ore
      - coal_ore
      - lapis_ore
      - mossy_cobblestone
      - obsidian
      - chest
      - diamond_ore
      - redstone_ore
      - clay
      - emerald_ore
      - ender_chest
      replacement-blocks:
      - stone
      - oak_planks
    generator-settings:
      flat-bedrock: false
    squid-spawn-height:
      maximum: 0.0
    baby-zombie-movement-modifier: 0.5
    viewdistances:
      no-tick-view-distance: 10
    wandering-trader:
      spawn-minute-length: 1200
      spawn-day-length: 24000
      spawn-chance-failure-increment: 25
      spawn-chance-min: 25
      spawn-chance-max: 75
    entity-per-chunk-save-limit:
      fireball: -1
      small_fireball: -1
      experience_orb: -1
      snowball: -1
      ender_pearl: -1
      arrow: -1
    mobs-can-always-pick-up-loot:
      zombies: false
      skeletons: false
    door-breaking-difficulty:
      zombie:
      - HARD
      vindicator:
      - NORMAL
      - HARD
    spawn-limits:
      monster: -1
      creature: -1
      underground_water_creature: -1
      water_creature: -1
      water_ambient: -1
      ambient: -1
    unsupported-settings:
      fix-invulnerable-end-crystal-exploit: true
    mob-effects:
      undead-immune-to-certain-effects: true
      spiders-immune-to-poison-effect: true
      immune-to-wither-effect:
        wither: true
        wither-skeleton: true
    tick-rates:
      sensor:
        villager:
          secondarypoisensor: 40
      behavior:
        villager:
          validatenearbypoi: -1
    feature-seeds:
      generate-random-seeds-for-all: false

```paper.yml

```server.properties
#Minecraft server properties
#Sat Dec 04 19:42:42 EST 2021
enable-jmx-monitoring=false
rcon.port=25575
level-seed=4488226861358040573
enable-command-block=false
gamemode=survival
enable-query=false
generator-settings=
level-name=easy
motd=Easy
query.port=25572
pvp=false
generate-structures=true
difficulty=easy
network-compression-threshold=256
max-tick-time=60000
require-resource-pack=false
max-players=100
use-native-transport=true
enable-status=true
online-mode=false
allow-flight=false
broadcast-rcon-to-ops=true
view-distance=5
max-build-height=256
server-ip=127.0.0.1
resource-pack-prompt=
allow-nether=true
server-port=25572
sync-chunk-writes=true
enable-rcon=false
server-name=easy
op-permission-level=4
prevent-proxy-connections=false
resource-pack=
entity-broadcast-range-percentage=100
player-idle-timeout=0
rcon.password=
force-gamemode=false
debug=false
rate-limit=0
hardcore=false
white-list=false
broadcast-console-to-ops=true
spawn-npcs=true
spawn-animals=true
snooper-enabled=true
function-permission-level=2
level-type=default
text-filtering-config=
spawn-monsters=true
enforce-whitelist=false
spawn-protection=0
resource-pack-sha1=
max-world-size=29999984
```server.properties

### Paper version

git-paper-391

Output here is after rollback to *stable* 388
[00:56:20 INFO]: This server is running Paper version git-Paper-388 (MC: 1.17.1) (Implementing API version 1.17.1-R0.1-SNAPSHOT) (Git: 22aaf91)
You are 3 version(s) behind
Download the new version at: https://papermc.io/downloads

### Other

screenshot of comparison of heapdumps taken at different intervals before server failed.

![image](https://user-images.githubusercontent.com/24461601/144735679-636c35af-492b-4857-8721-051cf3da1aa5.png)

heapdumps available upon request.

[electronicboy]

Over course of about 4 hours, grows to over 40G mem used

If your JVM is using that much memory, that's not the heap and cannot be the heap; There has been cases of reported native leaks somewhere over the years but nobody has been able to track down the source or even find a sane means of replicating it, nothing between 388 and 389 could have contributed towards a native memory leak

[waseric]

I hesitated to bother reporting because I haven't the expertise to adequately describe it. It's a new behavior in 391, impactful enough we had to revert. Any guidance on how to better characterize it appreciated, otherwise we can close. There's something there, but if I can't gather useful data, not much point in keeping this open.

Thanks for your time.

[e-im]

Thank you very much for the detailed and well-thought-through report; however, no changes between 388 and 391 could have possibly caused this. Something else must have changed here to cause this.

[supern64]

@waseric did you ever find the cause of this? our server is having the same issue.

[waseric]

@waseric did you ever find the cause of this? our server is having the same issue.

Sadly not. However, I do believe it is not related to this build, and most likely not to paper.

Sharing notes, in case it's at all helpful: It IS a native memory leak, as java heap is consistently within the allocated size. We've had a total of 5 of these events in the past 30 days - first of which pre-dated this build. User volume (high or low) is not a trigger. Specific users are not a trigger. We do not have good tools for analyzing/assessing native memory leak, but I can confirm that leaked file handles are not a symptom. And finally, only one of our servers has this issue, among many which all run the same software stack. Our owner has noted that the Jobs Reborn plugin has been cause of memory leaks in the past. I have no evidence to confirm that is the issue - it's one of the plugins unique to the affected server. That's my next area of focus.

[supern64]

We only have 1 server but we can also confirm the native leak, I haven't done any testing to single out what plugin is causing it however

[supern64]

we upgraded on dec 10 and the issues started happening on dec 14 so it may be another plugin...