GPU memory leak when opening/closing windows

ralgar commented 2 months ago

Regression?

Yes

System Info and Version

System/Version info

```sh Hyprland, built from branch at commit 0f594732b063a90d44df8c5d402d658f27471dfe (props: bump version to 0.43.0). Date: Sun Sep 8 09:48:21 2024 Tag: v0.43.0, commits: 5196 flags: (if any) System Information: System name: Linux Node name: gaming-desktop Release: 6.6.49-1-lts Version: #1 SMP PREEMPT_DYNAMIC Wed, 04 Sep 2024 13:18:51 +0000 GPU information: 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) (prog-if 00 [VGA controller]) NVRM version: NVIDIA UNIX x86_64 Kernel Module 560.35.03 Fri Aug 16 21:39:15 UTC 2024 os-release: NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo plugins: ======Config-Start====== Config File: /home/ralgar/.config/hypr/hyprland.conf: Read Succeeded # ########################################################################### # # # ██╗ ██╗██╗ ██╗██████╗ ██████╗ ██╗ █████╗ ███╗ ██╗██████╗ # # # # ██║ ██║╚██╗ ██╔╝██╔══██╗██╔══██╗██║ ██╔══██╗████╗ ██║██╔══██╗ # # # # ███████║ ╚████╔╝ ██████╔╝██████╔╝██║ ███████║██╔██╗ ██║██║ ██║ # # # # ██╔══██║ ╚██╔╝ ██╔═══╝ ██╔══██╗██║ ██╔══██║██║╚██╗██║██║ ██║ # # # # ██║ ██║ ██║ ██║ ██║ ██║███████╗██║ ██║██║ ╚████║██████╔╝ # # # # ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝╚═╝ ╚═══╝╚═════╝ # # # ########################################################################### # # Refer to https://wiki.hyprland.org for more information. # Keep the primary monitor at its highest refresh rate. monitor = DP-1,preferred,0x0,1 # Keep a large buffer zone for the auxilliary, so that the mouse doesn't traverse displays. # Mouse can be switched to the aux display using workspace keybinds. monitor = DP-2,highres,3840x0,1 input { # Keyboard Options numlock_by_default = 1 # Mouse/Touchpad Options sensitivity = 0.0 # Mouse sensitivity (range -1.0 to 1.0) accel_profile = flat # Mouse acceleration (adaptive, flat, or custom) touchpad { disable_while_typing = true } follow_mouse = 1 # Window focus follows mouse } general { gaps_in=5 gaps_out=10 border_size=2 col.active_border=0xff5c6370 col.inactive_border=0xff3e4452 } misc { disable_hyprland_logo = true disable_splash_rendering = true } decoration { rounding=9 # Use an odd value here for perfect symmetry. blur { enabled=0 # 0 to disable, 1 to enable. size=3 # Minimum 1, high values (5+) may cause artifacts. passes=1 # Minimum 1, more passes = more resource intensive. } drop_shadow = true shadow_range = 10 shadow_render_power = 3 col.shadow = rgba(1a1a1aee) } animations { enabled = true # Some default animations, see https://wiki.hyprland.org/Configuring/Animations/ for more bezier = myBezier, 0.05, 0.9, 0.1, 1.05 animation = windows, 1, 7, myBezier animation = windowsOut, 1, 7, default, popin 80% animation = border, 1, 10, default animation = borderangle, 1, 8, default animation = fade, 1, 7, default animation = workspaces, 1, 6, default } dwindle { pseudotile = false # Initialize with pseudotiling off force_split = 2 # Always split towards the right/bottom preserve_split = true # You probably want this on } # #################### # # # WINDOW RULES # # # #################### # windowrule=workspace 2,^(firefox)$ windowrule=fullscreen,^(gamescope)$ # Inhibit idle when the primary display has a fullscreen window windowrulev2 = idleinhibit focus,monitor: 1,fullscreen: 1 # Fullscreen Discord on the auxilliary display windowrule=monitor 2,^(discord)$ windowrule=fullscreen,^(discord)$ # ################ # # # KEYBINDS # # # ################ # # General bind=SUPER,Return,exec,alacritty bind=SUPER,Q,killactive, bind=SUPERSHIFT,Q,exit, bind=SUPER,E,exec,wofi bind=SUPER,V,togglefloating, bind=SUPER,L,exec,hyprlock bind=SUPER,F,fullscreen # Window focus bind=SUPER,left,movefocus,l bind=SUPER,right,movefocus,r bind=SUPER,up,movefocus,u bind=SUPER,down,movefocus,d # Window Move and Resize bindm=SUPER,mouse:272,movewindow # Left Mouse bindm=SUPER,mouse:273,resizewindow # Right Mouse # Workspaces bind=SUPER,1,workspace,1 bind=SUPER,2,workspace,2 bind=SUPER,3,workspace,3 bind=SUPER,4,workspace,4 bind=SUPER,5,workspace,5 bind=SUPER,6,workspace,6 bind=SUPER,7,workspace,7 bind=SUPER,8,workspace,8 bind=SUPER,9,workspace,9 bind=SUPER,0,workspace,10 bind=SUPERSHIFT,1,movetoworkspace,1 bind=SUPERSHIFT,2,movetoworkspace,2 bind=SUPERSHIFT,3,movetoworkspace,3 bind=SUPERSHIFT,4,movetoworkspace,4 bind=SUPERSHIFT,5,movetoworkspace,5 bind=SUPERSHIFT,6,movetoworkspace,6 bind=SUPERSHIFT,7,movetoworkspace,7 bind=SUPERSHIFT,8,movetoworkspace,8 bind=SUPERSHIFT,9,movetoworkspace,9 bind=SUPERSHIFT,0,movetoworkspace,10 # Media keys bind=,XF86AudioPlay,exec,mpc toggle bind=,XF86AudioMute,exec,amixer -q set Master toggle binde=,XF86AudioLowerVolume,exec,amixer -q set Master 5%- binde=,XF86AudioRaiseVolume,exec,amixer -q set Master 5%+ # Screenshots and Recordings bind=,Print,exec,grim -o $(hyprctl -j activeworkspace | jq -r .monitor) bind=SUPERSHIFT,R,exec,save-replay.sh # ####################### # # # Workspace Rules # # # ####################### # # Bind workspace 0 to auxilliary monitor workspace = 1, monitor:DP-1 workspace = 2, monitor:DP-1 workspace = 3, monitor:DP-1 workspace = 4, monitor:DP-1 workspace = 5, monitor:DP-1 workspace = 10, monitor:DP-2 # ################# # # # Autostart # # # ################# # exec-once=hyprpaper exec-once=waybar exec-once=mpd exec-once=gammastep exec-once=mako exec-once=hypridle exec-once=yubikey-touch-detector --libnotify #exec-once=sh -c "ssh-agent -D -a $XDG_RUNTIME_DIR/ssh-agent.socket" ======Config-End======== ```

Description

When a native Wayland or Xwayland window is opened and then closed, total Hyprland vRAM usage for the session is permanently increased. I've seen it reach nearly a gigabyte over the course of a day.

I'm not sure if it's related, but the Xwayland server also seems to be leaking GPU memory in the same manner, often reaching nearly half a gig itself after using Steam and playing games for a while.

I first noticed the behavior around v0.41 and it's been consistent ever since, I just hadn't gotten around to opening an issue until now. If you can't repro, or need me to bisect, let me know and I'll get to it when I have a moment.

How to reproduce

Simply open and close any application window while monitoring Hyprland's GPU memory usage. Usage increases by about 10-20MiB every time.

For the Xwayland server, do the same but with an Xwayland app such as Steam. You don't need to exit Steam each time, just open and close the window.

Crash reports, logs, images, videos

Sun Sep  8 11:49:04 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1060 6GB    Off |   00000000:01:00.0  On |                  N/A |
|  0%   59C    P0             23W /  150W |    1457MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      9023      G   Hyprland                                      787MiB |
|    0   N/A  N/A      9157      G   Xwayland                                      440MiB |
|    0   N/A  N/A     10092      G   /usr/lib/firefox/firefox                      103MiB |
|    0   N/A  N/A     13492      G   ...local/share/Steam/ubuntu12_32/steam          2MiB |
|    0   N/A  N/A     13867      G   ./steamwebhelper                               27MiB |
|    0   N/A  N/A     14076      G   alacritty                                      40MiB |
+-----------------------------------------------------------------------------------------+

3ri4nG0ld commented 2 months ago

imagen

._.

vaxerski commented 2 months ago

curious. Can you bisect this? I assume 0.42 did not exhibit this.

Greatz08 commented 2 months ago

same here i am using hyprland latest 0.43 on arch based linux garuda with latest kernel normal linux kernel 6.10 and i tested by opening and closing kitty again and again and again and it keeps on consuming more and more and more you can test yourself and will observe same behavior so yeah please fix this as soon as possible . Because at this rate i wont be able to run my local llm and keep closing and opening stuff :-((

ralgar commented 2 months ago

curious. Can you bisect this? I assume 0.42 did not exhibit this.

The issue was definitely present in 0.42, and I'm pretty sure 0.41 as well. I just didn't get around to opening an issue.

I'll see about bisecting it this morning.

Edit: 0.41.2 is fine actually. No xwayland or Hyprland leaks there. It's just 0.42+. Going to start bisecting now.

ralgar commented 2 months ago

This is what I landed on when bisecting for the Hyprland process leak:

2d552fbaa25f1457c3819521a2750dd30820271b is the first bad commit
commit 2d552fbaa25f1457c3819521a2750dd30820271b
Author: Vaxry <vaxry@vaxry.net>
Date:   Wed Aug 7 18:54:45 2024 +0200

    renderer: fixup nvidia driver version checks

 src/render/Renderer.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

https://github.com/hyprwm/Hyprland/commit/2d552fbaa25f1457c3819521a2750dd30820271b

I noticed that this commit also increased idle/startup vRAM usage from 137MB to 227MB.

Edit: After checking out the relevant code, it looks like this is related to explicit sync. If I set render.explicit_sync = 0, then the leak is no longer present, and the startup vRAM usage returns to 137MB.

As for the Xwayland leak, it does appear to be a Hyprland issue also, but it was introduced in a different commit. Would you like me to open a separate issue for that one @vaxerski? I'd be happy to bisect that as well, but it probably won't be today.

Edit: This one is also resolved by disabling explicit sync.

vaxerski commented 2 months ago

maybe thats an nvidia bug? I mean, it doesnt seem to leak on intel/amdeeznuts

ralgar commented 2 months ago

maybe thats an nvidia bug? I mean, it doesnt seem to leak on intel/amdeeznuts

Yeah, that seems likely then. If that's the case, we would probably expect to see leaks in other OpenGL (and possibly Vulkan) applications as well, no?

Edit: Yeah, not exactly inspiring confidence here lol. I swear, an AMD GPU is the next thing on my shopping list.

vaxerski commented 2 months ago

closing then, good luck with nvidia I guess

Greatz08 commented 2 months ago

dayumm

maybe thats an nvidia bug? I mean, it doesnt seem to leak on intel/amdeeznuts

Yeah, that seems likely then. If that's the case, we would probably expect to see leaks in other OpenGL (and possibly Vulkan) applications as well, no?

Edit: Yeah, not exactly inspiring confidence here lol. I swear, an AMD GPU is the next thing on my shopping list.

dayummm that is the reason again nvidia shi*** driver issues . I can confirm that it doesnt happen in amd igpu atleast

ralgar commented 2 months ago

For anyone else coming across this - I've noticed absolutely no functional difference when setting render.explicit_sync = 0 in my hyprland.conf. There's no flicker in apps or games, everything works just fine, so it would seem to be an acceptable solution until Nvidia fixes this.

To play devil's advocate for a minute - 560 is still a 'new feature' branch at present, not the production branch. Hopefully this issue will be ironed out soon.

3ri4nG0ld commented 2 months ago

My new AMD PC arrives on the 26th, fuck NVIDIA

vfosterm commented 2 weeks ago

Hey everyone. Found a fix for the NVIDIA VRAM leak (besides buying an AMD GPU :D) https://github.com/NVIDIA/egl-wayland/issues/126#issuecomment-2379945259

just replace kwin_wayland with Hyprland in the file. I'll upload my working one here as well. Just place it in /etc/nvidia/nvidia-application-profiles-rc.d and logout and back in again. 50-limit-free-buffer-pool-in-wayland-compositors.json

hyprwm / Hyprland