GPU memory leak in editor

mnn commented 4 years ago

Godot version: 3.1.2.stable.official

OS/device including version: Linux 64b

Issue description: GPU memory in editor is not being released after closing files in which graphical resources are used (possibly related to AnimatedSprite).

Example (sequential), values are GPU memory measured by nvidia-smi:

in project manager: 429MiB
after opening project with empty scene: 618MiB
after opening scene with animation: 918MiB
after closing scene with animation and waiting 10 seconds: 918MiB (results sometimes wary slightly, e.g. sometimes I got 961MiB, but that's still much higher compared to 618MiB)

Steps to reproduce:

Create a new scene with AnimatedSprite
Check GPU mem - baseline
Create new frames in inspector
Copy non-tiny animation files (e.g. 200 of PNGs each ~1MB) to project
Drag and drop to animation (bottom panel)
Check GPU mem (should be higher for Godot editor process)
Close scene
Check GPU mem - doesn't go back to baseline

Minimal reproduction project: Well, not very minimal when judging by size :smile:. It's too big for GH, so I had to upload it somewhere else: https://mega.nz/#!sAI0VKQI!QAW42ZeR-IjHjBeCDmhLH_9Gr7wi83oy3GjFWMmK7HM

mnn commented 4 years ago

I forgot to mention, this has become quite a serious issue for us, because after opening Character scene (= many animations), editor skyrockets to 5GiB and never goes down even after closing Character scene. When editor is in this half-broken state, I can't run the game well, because I am hitting maximal GPU memory. The FPS drops down and is rather unstable, dropping even to single digits. Load times of the game also increase significantly, taking approx. 4-5 times more than usual. We haven't even finished first level, I dread of the future... Only reliable workaround is closing project and re-opning it which is quite cumbersome because I lose cursor positions in all opened scripts. And of course this isn't even possible when working on the Character scene.

Because of this, I would ask to consider tagging this issue with some higher priority, it would be nice if this would get fixed in 3 branch.

clayjohn commented 4 years ago

I will take a look into this when I am back home.

In the meantime, can you let me know:

if this issue exists for GLES3 and GLES2 or just one or the other,
what GPU you are using + what version of the drivers you are using (may be worth updating drivers),
and please let me know if you can reproduce on 3.2beta4.

Thanks in advance.

kyleguarco commented 4 years ago

I couldn't reproduce the issue using the project and steps provided. I tried it with GLES2 and GLES3

Godot Versions: 3.2.beta.custom_build.85fa17d3c (master, commit 85fa17d3c) 3.1.2.stable.custom_build.ec02a81 (commit ec02a81)

Graphics: Intel UHD Graphics 620 (Whiskey Lake) intel-gpu-tools used for monitoring Driver: xf86-video-intel 1:2.99.917+899+gf66d3954 OS: Arch Linux, kernel 5.4.6-arch3-1

edit Provided drivers and renderers

mnn commented 4 years ago

if this issue exists for GLES3 and GLES2 or just one or the other,

In both. In one test (GLES2) I saw GPU memory being slowly released, but not in full, only ~10% released of what should have been.

what GPU you are using + what version of the drivers you are using (may be worth updating drivers),

NVidia 2080Ti, 440.36.

full system info

$ inxi -Fxxxz
System:    Host: xxx Kernel: 5.4.2-1-MANJARO x86_64 bits: 64 compiler: gcc v: 9.2.0 Desktop: i3 4.17.1 info: i3bar dm: SDDM
           Distro: Manjaro Linux
Machine:   Type: Desktop Mobo: ASRock model: X570 Phantom Gaming X serial:  UEFI: American Megatrends v: P1.80
           date: 08/08/2019
CPU:       Topology: 8-Core model: AMD Ryzen 7 3800X bits: 64 type: MT MCP arch: Zen L2 cache: 4096 KiB
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 124850
           Speed: 2200 MHz min/max: 2200/3900 MHz boost: enabled Core speeds (MHz): 1: 2610 2: 1932 3: 1934 4: 1909 5: 2049
           6: 1869 7: 2023 8: 2273 9: 2111 10: 1920 11: 2024 12: 2086 13: 2022 14: 2739 15: 2825 16: 2123
Graphics:  Device-1: NVIDIA TU102 [GeForce RTX 2080 Ti Rev. A] vendor: ASUSTeK driver: nvidia v: 440.36 bus ID: 10:00.0
           chip ID: 10de:1e07
           Display: x11 server: X.Org 1.20.6 driver: nvidia resolution: 1920x1080~60Hz, 3840x2160~60Hz, 1920x1080~60Hz
           OpenGL: renderer: GeForce RTX 2080 Ti/PCIe/SSE2 v: 4.6.0 NVIDIA 440.36 direct render: Yes
Audio:     Device-1: NVIDIA TU102 High Definition Audio vendor: ASUSTeK driver: snd_hda_intel v: kernel bus ID: 10:00.1
           chip ID: 10de:10f7
           Device-2: Advanced Micro Devices [AMD] Starship/Matisse HD Audio vendor: ASRock driver: snd_hda_intel v: kernel
           bus ID: 12:00.4 chip ID: 1022:1487
           Sound Server: ALSA v: k5.4.2-1-MANJARO
Network:   Device-1: Intel Wi-Fi 6 AX200 driver: iwlwifi v: kernel bus ID: 08:00.0 chip ID: 8086:2723
           IF: wlp8s0 state: down mac: 
           Device-2: Intel I211 Gigabit Network vendor: ASRock driver: igb v: 5.6.0-k port: e000 bus ID: 0a:00.0
           chip ID: 8086:1539
           IF: enp10s0 state: up speed: 100 Mbps duplex: half mac: 
           Device-3: Realtek RTL8125 2.5GbE vendor: ASRock driver: r8169 v: kernel port: d000 bus ID: 0c:00.0
           chip ID: 10ec:8125
           IF: enp12s0 state: down mac: 
Drives:    Local Storage: total: 2.05 TiB used: 1.65 TiB (80.7%)
           ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO Plus 250GB size: 232.89 GiB speed: 31.6 Gb/s lanes: 4
           serial:  rev: 1B2QEXM7 scheme: GPT
           ID-2: /dev/nvme1n1 vendor: Samsung model: SSD 970 EVO Plus 250GB size: 232.89 GiB speed: 31.6 Gb/s lanes: 4
           serial:  rev: 1B2QEXM7 scheme: GPT
           ID-3: /dev/nvme2n1 vendor: Samsung model: SSD 970 EVO Plus 250GB size: 232.89 GiB speed: 31.6 Gb/s lanes: 4
           serial:  rev: 1B2QEXM7 scheme: GPT
           ID-4: /dev/sda vendor: Western Digital model: WD20EZRZ-00Z5HB0 size: 1.82 TiB speed: 6.0 Gb/s rotation: 5400 rpm
           serial:  rev: 0A80 scheme: GPT
Partition: ID-1: / size: 219.27 GiB used: 181.24 GiB (82.7%) fs: ext4 dev: /dev/dm-0
           ID-2: swap-1 size: 8.80 GiB used: 3.2 MiB (0.0%) fs: swap dev: /dev/dm-1
Sensors:   System Temperatures: cpu: 56.9 C mobo: 41.0 C gpu: nvidia temp: 43 C
           Fan Speeds (RPM): fan-1: 0 fan-2: 1125 fan-3: 0 fan-4: 776 fan-5: 773 fan-6: 815 fan-7: 897 gpu: nvidia fan: 34%
Info:      Processes: 376 Uptime: 13h 30m Memory: 31.29 GiB used: 6.62 GiB (21.2%) Init: systemd v: 242 Compilers: gcc: 9.2.0
           alt: 7/8 Shell: zsh v: 5.7.1 running in: terminology inxi: 3.0.37

The driver is quite new (a month old I think). I'll update as soon as I can, hopefully on Friday. (Occasionally there are issues with GPU drivers and I don't want to end up in a state of not being able to work for a day job.)

and please let me know if you can reproduce on 3.2beta4.

GLES2:

empty scene: 500MiB
scene with anim: 800MiB
after closing anim scene: 797MiB

GLES3:

empty scene: 1067MiB
scene with anim: 1367MiB
after closing anim scene: 1364MiB

After GLES switch I always closed and reopned project, just to be sure. GPU mem usage doesn't go down even after few minutes.

clayjohn commented 4 years ago

Thanks! I have an NVidia GPU as well on my desktop so I'll try to reproduce as well. It looks like it may be an NVidia/driver issue.

mnn commented 4 years ago

I hope you'll be successful, because I don't have a clue what else I could provide in order to debug this (my experience with C++ is rather limited and I don't know much about low-level graphical stuff nor internals of Godot).

I am not sure how inxi reports open-source drivers, but I am using those proprietary ones provided by NVidia.

kyleguarco commented 4 years ago

@clayjohn I was able to reproduce the issue on another PC running Windows 10, using GLES2 and GLES3

Godot Version: 3.2.beta4.official (commit d1bce5c67)

OS: Windows 10 1903 (build 10.0.18362) Graphics: NVIDIA GeForce GTX 1050, official driver version 436.48 (26.21.14.3648) GPU-Z used for monitoring

edit Forgot the renderers again

mnn commented 4 years ago

Sorry, got to it a bit later. Updated to 440.44 - most recent in Manjaro. I have tried it with the beta 4 (bold values on each line should be same/close):

GLES3: 1067MiB, 1367MiB, 1364MiB
GLES2: 500MiB, 800MiB, 797MiB

It seems to me exactly same as with the older drivers.

mnn commented 4 years ago

Tried 3.2 stable, problem persists (values in MiB, bold values on same line should be equal/close):

GLES3: 1059, 1359, 1356
GLES2: 487, 787, 787

KoBeWi commented 3 years ago

Can anyone still reproduce this bug in Godot 3.2.3 or any later release?

mnn commented 3 years ago

Just tested the repro project, it's still present in 3.2.3 (official).

GLES3: 1056, 1356, 1356
GLES2: 483, 783, 765

Note: To quickly get VRAM consumption of Godot editor on an NVidia card on Linux you can use:

$ nvidia-smi pmon -s um -c 1 | grep -Ei '#|godot'
# gpu        pid  type    sm   mem   enc   dec    fb   command
# Idx          #   C/G     %     %     %     %    MB   name
    0     177234     G     -     -     -     -   765   Godot_v3.2.3-st

Drivers: 455.45.01 (latest in Manjaro, I believe there are no newer stable drivers)

godotengine / godot

GPU memory leak in editor #34732