search

TigerVNC / tigervnc

High performance, multi-platform VNC client and server
https://tigervnc.org
GNU General Public License v2.0
5.05k stars 931 forks source link

XVNC Memory Leak #1553

Open cantenna opened 1 year ago

cantenna commented 1 year ago

Shortly after starting a TigerVNC sesion, XVNC process continues to consume all available system memory. This bug is applicable to XVNC as behavior is present with any vnc application which utilities XVNC backend

install tigervnc-1.12.0-3 and start vncserver session execute "top" command to observe XVNC memory leak

Expected behavior Previously memory would rarely exceed 0.5%

Screenshots memory leak 2

Client

Server

If i restore my system to a backup taken 30 days prior issue is not present. Issue must therefore must be an package update.

Does not appear to be related to packages; Xorg, Mesa, Vulkan, Python, QT5, Intel, Nvidia

In an effort to troubleshoot, I upgraded these package independently and held back other updates and the issue was not present.

--- Want to back this issue? **[Post a bounty on it!](https://app.bountysource.com/issues/113863922-xvnc-memory-leak?utm_campaign=plugin&utm_content=tracker%2F3557444&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://app.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F3557444&utm_medium=issues&utm_source=github).
CendioOssman commented 1 year ago
  • VNC server: [have tried TigerVNC, TurboVNC & XRDP]

Does that mean you are seeing the memory leak with not just TigerVNC's Xvnc, but with other servers as well?

In an effort to troubleshoot, I upgraded these package independently and held back other updates and the issue was not present.

And if you upgrade only TigerVNC and nothing else?

samueldy commented 1 year ago

Want to report that I'm seeing this, too.

Client:

Server:

Steps to reproduce:

  1. Boot server into kernel 6.0.10-1-MANJARO.
  2. Start the VNC server using the provided Systemd unit (e.g., sudo systemctl start vncserver@:<DISPLAY>.service).
  3. Connect to VNC session over SSH from client computer using TigerVNC Viewer. This works successfully, and I am able to do some VNC work as usual.
  4. Disconnect the client from the server by closing the client TigerVNC Viewer and logging out of the SSH session.
  5. Some time later, log into the server from the client using SSH. Observe using systemctl status vncserver@:<DISPLAY>.service or journalctl -xeu vncserver:@<DISPLAY>.service that the VNC service has been forcefully killed. Typical journalctl output:
Dec 03 12:11:06  systemd[1]: vncserver@:<DISPLAY>.service: Main process exited, code=killed, status=9/KILL
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://forum.manjaro.org/c/support
░░ 
░░ An ExecStart= process belonging to unit vncserver@:<DISPLAY>.service has exited.
░░ 
░░ The process' exit code is 'killed' and its exit status is 9.

The VNC server appears to get killed (presumably by the OOM reaper) usually about 1 to 2 hours after I disconnect from the VNC session.

samueldy commented 1 year ago

Update: I tracked the memory of the Xvnc process over time and did not observe any spike in the Xvnc process memory usage before it was killed.

So I tried booting the server into Manjaro's 5.15.81-1-MANJARO #1 kernel to see if that would work. vncserver crashed again, and I noticed journalctl entries from around that time:

journalctl scrollback ``` Dec 04 15:53:37 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@4-88046-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Dec 04 15:53:37 kernel: audit: type=1130 audit(1670187217.602:268): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@4-88046-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Dec 04 15:54:12 kernel: panel-9-pulseau: page allocation failure: order:0, mode:0x1100cca(GFP_HIGHUSER_MOVABLE), nodemask=(null),cpuset=user.slice,mems_allowed=0 Dec 04 15:54:13 kernel: CPU: 6 PID: 66562 Comm: panel-9-pulseau Tainted: P OE 5.15.81-1-MANJARO #1 342159d7ac76ce9002928cd2f94e5471a7c4bb76 Dec 04 15:54:13 kernel: Hardware name: Dec 04 15:54:13 kernel: Call Trace: Dec 04 15:54:13 kernel: Dec 04 15:54:13 kernel: dump_stack_lvl+0x45/0x63 Dec 04 15:54:13 kernel: warn_alloc+0x164/0x190 Dec 04 15:54:13 kernel: __alloc_pages_slowpath.constprop.0+0xd8d/0xdb0 Dec 04 15:54:13 kernel: __alloc_pages+0x20f/0x240 Dec 04 15:54:13 kernel: alloc_pages_vma+0x8b/0x310 Dec 04 15:54:13 kernel: shmem_alloc_page+0x97/0xf0 Dec 04 15:54:13 kernel: shmem_alloc_and_acct_page+0x79/0x1d0 Dec 04 15:54:13 kernel: shmem_getpage_gfp+0x4d7/0x920 Dec 04 15:54:13 kernel: shmem_fault+0x93/0x280 Dec 04 15:54:13 kernel: __do_fault+0x33/0x120 Dec 04 15:54:13 kernel: __handle_mm_fault+0xd83/0x1550 Dec 04 15:54:13 kernel: handle_mm_fault+0xd1/0x2c0 Dec 04 15:54:13 kernel: __get_user_pages+0x218/0x6b0 Dec 04 15:54:13 kernel: get_dump_page+0xbb/0x310 Dec 04 15:54:13 kernel: dump_user_range+0x76/0xf0 Dec 04 15:54:13 kernel: elf_core_dump+0xd3a/0xf40 Dec 04 15:54:13 kernel: do_coredump+0x10c8/0x17f0 Dec 04 15:54:13 kernel: get_signal+0x119/0x8f0 Dec 04 15:54:13 kernel: ? complete_signal+0xe0/0x2a0 Dec 04 15:54:13 kernel: arch_do_signal_or_restart+0xdd/0x740 Dec 04 15:54:13 kernel: exit_to_user_mode_prepare+0x101/0x190 Dec 04 15:54:13 kernel: irqentry_exit_to_user_mode+0x5/0x20 Dec 04 15:54:13 kernel: asm_exc_int3+0x35/0x40 Dec 04 15:54:13 kernel: RIP: 0033:0x7f8d22c75048 Dec 04 15:54:13 kernel: Code: Unable to access opcode bytes at RIP 0x7f8d22c7501e. Dec 04 15:54:13 kernel: RSP: 002b:00007ffe99302700 EFLAGS: 00000202 Dec 04 15:54:13 kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 Dec 04 15:54:13 kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00000000000003a0 Dec 04 15:54:13 kernel: RBP: 0000000000000004 R08: 00005575b64b06c0 R09: 0000000000000073 Dec 04 15:54:13 kernel: R10: 0000000000000000 R11: 6bd84eda7cac9bee R12: 0000000000000000 Dec 04 15:54:13 kernel: R13: 00007ffe99302750 R14: 00007f8d22d55d00 R15: 00007f8d22d55d70 Dec 04 15:54:13 kernel: Dec 04 15:54:13 kernel: Mem-Info: Dec 04 15:54:13 kernel: active_anon:380899 inactive_anon:3071381 isolated_anon:32 active_file:155 inactive_file:0 isolated_file:0 unevictable:152 dirty:67 writeback:10 slab_reclaimable:71195 slab_unreclaimable:59898 mapped:3032809 shmem:3044047 pagetables:26480 bounce:0 kernel_misc_reclaimable:0 free:33185 free_pcp:1 free_cma:0 ```

and later on:

More journalctl scrollback ``` Dec 04 15:54:13 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=system-systemd\x2dcoredump.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-4.scope,task=panel-9-pulseau,pid=66562,uid=1000 Dec 04 15:54:13 kernel: Out of memory: Killed process 66562 (panel-9-pulseau) total-vm:66818668kB, anon-rss:14204kB, file-rss:4kB, shmem-rss:11918380kB, UID:1000 pgtables:23612kB oom_score_adj:0 ```

followed by SIGKILLs of everything in the VNC session.

Looks like the Pulseaudio volume control widget in my XFCE4 panel isn't happy.

When I use a VNC session on this server, I usually forward my client's Pulseaudio server over SSH as detailed here. Previously, disconnecting from the SSH session and leaving the server's Pulseaudio applications with a non-responsive port to which to send their audio wouldn't cause any problems, and when I reconnected to the VNC session the apps and the panel Pulseaudio widget would work just fine. But now it appears the widget or something in the Pulseaudio subsystem is eventually running out of memory when it can't access a functional Pulseaudio server on a port.

Will try another session without Pulseaudio forwarding and see where that lands me.

CendioOssman commented 1 month ago

@cantenna, @samueldy, are you still seeing this issue?

piotr-dobrogost commented 1 month ago

Incidentally I came here after oom-killer started killing Xvnc after I upgraded from Fedora 39 to Fedora 40 (KDE spin):

sie 09 15:34:44 demon kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=gpm.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-34.scope,task=Xvnc,pid=99187,uid=1000
sie 09 15:34:44 demon kernel: Out of memory: Killed process 99187 (Xvnc) total-vm:12491216kB, anon-rss:6600672kB, file-rss:104kB, shmem-rss:48192kB, UID:1000 pgtables:24372kB oom_score_adj:0

However according to https://packages.fedoraproject.org/pkgs/tigervnc/tigervnc-server/ both Fedora 39 and Fedora 40 have the same version of TigerVNC – 1.13.1

It looks like observing Memory entry from systemctl status for vncserver service would allow to see if Xnvc's memory consumption raises:

[piotr@demon]~% sudo systemctl status vncserver@:1.service
● vncserver@:1.service - Remote desktop service (VNC)
     Loaded: loaded (/etc/systemd/system/vncserver@.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Fri 2024-08-09 15:38:11 CEST; 55min ago
    Process: 115646 ExecStartPre=/usr/libexec/vncsession-restore :1 (code=exited, status=0/SUCCESS)
    Process: 115658 ExecStart=/usr/libexec/vncsession-start :1 (code=exited, status=0/SUCCESS)
   Main PID: 115666 (vncsession)
      Tasks: 0 (limit: 9349)
     Memory: 3.3M (peak: 4.4M)
        CPU: 37ms
     CGroup: /system.slice/system-vncserver.slice/vncserver@:1.service
             ‣ 115666 /usr/sbin/vncsession piotr :1

as the process tree looks like this:

[piotr@demon]~% pstree -p -s 115666
systemd(1)───vncsession(115666)───xinit(115667)─┬─Xvnc(115673)
                                                └─startplasma-x11(115675)─┬─ssh-agent(115744)
                                                                          └─{startplasma-x11}(115772)

I think this info gets lost when the process is killed and the service becomes inactive so it's necessary to observe it for a longer period of time saving the status like this:

#!/usr/bin/env bash

while systemctl status vncserver@:1.service 2>&1 > /dev/null
do
   systemctl status vncserver@:1.service 2>&1 | tee -a /tmp/xvnc-status.txt
   sleep 60
done

However there might be a case of a sudden spike in memory usage leading to oom-killer killing Xvnc and this type of periodic monitoring won't catch this…

Any hints how to go about diagnosing this would be appreciated.

EDIT

Using the following script

#!/usr/bin/env bash

while true
do
   date | tee -a /tmp/memory.txt
   ps aux --sort=-%mem | head -n10 2>&1 | tee -a /tmp/memory.txt
   echo -e "-----\n" | tee -a /tmp/memory.txt
   sleep 60
done

I managed to observe sudden raise in Xvnc's memory consumption:

sob, 10 sie 2024, 13:17:03 CEST
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
piotr     136778 10.9 24.8 8866448 2001236 ?     Ssl  12:46   3:20 /home/piotr/apps/pycharm-2024.1.4/jbr/bin/java -classpath /home/piotr/apps/pycharm-2024.1.4/lib/platform-loader.jar:/home/piotr/apps/pycharm-2024.1.4/lib/util-8.jar:/home/piotr/apps/pycharm-2024.1.4/lib/util.jar:/home/piotr/apps/pycharm-2024.1.4/lib/app-client.jar:/home/piotr/apps/pycharm-2024.1.4/lib/util_rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/product.jar:/home/piotr/apps/pycharm-2024.1.4/lib/opentelemetry.jar:/home/piotr/apps/pycharm-2024.1.4/lib/app.jar:/home/piotr/apps/pycharm-2024.1.4/lib/product-client.jar:/home/piotr/apps/pycharm-2024.1.4/lib/modules.jar:/home/piotr/apps/pycharm-2024.1.4/lib/lib-client.jar:/home/piotr/apps/pycharm-2024.1.4/lib/stats.jar:/home/piotr/apps/pycharm-2024.1.4/lib/jps-model.jar:/home/piotr/apps/pycharm-2024.1.4/lib/external-system-rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/rd.jar:/home/piotr/apps/pycharm-2024.1.4/lib/bouncy-castle.jar:/home/piotr/apps/pycharm-2024.1.4/lib/protobuf.jar:/home/piotr/apps/pycharm-2024.1.4/lib/intellij-test-discovery.jar:/home/piotr/apps/pycharm-2024.1.4/lib/intellij-coverage-agent-1.0.744.jar:/home/piotr/apps/pycharm-2024.1.4/lib/forms_rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/lib.jar:/home/piotr/apps/pycharm-2024.1.4/lib/externalProcess-rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/groovy.jar:/home/piotr/apps/pycharm-2024.1.4/lib/annotations.jar:/home/piotr/apps/pycharm-2024.1.4/lib/async-profiler.jar:/home/piotr/apps/pycharm-2024.1.4/lib/grpc.jar:/home/piotr/apps/pycharm-2024.1.4/lib/jsch-agent.jar:/home/piotr/apps/pycharm-2024.1.4/lib/junit4.jar:/home/piotr/apps/pycharm-2024.1.4/lib/trove.jar -XX:ErrorFile=/home/piotr/java_error_in_pycharm_%p.log -XX:HeapDumpPath=/home/piotr/java_error_in_pycharm_.hprof -Xms256m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -XX:+IgnoreUnrecognizedVMOptions -ea -Dsun.io.useCanonCaches=false -Dsun.java2d.metal=true -Djbr.catch.SIGABRT=true -Djdk.http.auth.tunneling.disabledSchemes="" -Djdk.attach.allowAttachSelf=true -Djdk.module.illegalAccess.silent=true -Dkotlinx.coroutines.debug=off -XX:CICompilerCount=2 -XX:ReservedCodeCacheSize=512m -XX:CompileCommand=exclude,com/intellij/openapi/vfs/impl/FilePartNodeRoot,trieDescend -XX:SoftRefLRUPolicyMSPerMB=50 -Dide.show.tips.on.startup.default.value=false -Dsun.tools.attach.tmp.only=true -Dawt.lock.fair=true -Djava.net.preferIPv4Stack=true -Dawt.useSystemAAFontSettings=lcd -Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine -Dide.managed.by.toolbox=/home/piotr/.local/share/JetBrains/Toolbox/bin/jetbrains-toolbox -Dtoolbox.notification.token=8ae28aa5-1d2d-4c8b-b806-98fa259920d7 -Dtoolbox.notification.portFile=/home/piotr/.cache/JetBrains/Toolbox/ports/397276050.port -Dawt.toolkit.name=WLToolkit -Djb.vmOptionsFile=/home/piotr/.config/JetBrains/PyCharm2024.1/pycharm64.vmoptions -Djava.system.class.loader=com.intellij.util.lang.PathClassLoader -Didea.vendor.name=JetBrains -Didea.paths.selector=PyCharm2024.1 -Djna.boot.library.path=/home/piotr/apps/pycharm-2024.1.4/lib/jna/amd64 -Dpty4j.preferred.native.folder=/home/piotr/apps/pycharm-2024.1.4/lib/pty4j -Djna.nosys=true -Djna.noclasspath=true -Dintellij.platform.runtime.repository.path=/home/piotr/apps/pycharm-2024.1.4/modules/module-descriptors.jar -Didea.platform.prefix=Python -Dsplash=true -Daether.connector.resumeDownloads=false --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.ref=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/jdk.internal.vm=ALL-UNNAMED --add-opens=java.
base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.fs=ALL-UNNAMED --add-opens=java.base/sun.security.ssl=ALL-UNNAMED --add-opens=java.base/sun.security.util=ALL-UNNAMED --add-opens=java.base/sun.net.dns=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing.plaf.gtk=ALL-UNNAMED --add-opens=java.desktop/java.awt=ALL-UNNAMED --add-opens=java.desktop/java.awt.dnd.peer=ALL-UNNAMED --add-opens=java.desktop/java.awt.event=ALL-UNNAMED --add-opens=java.desktop/java.awt.image=ALL-UNNAMED --add-opens=java.desktop/java.awt.peer=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED --add-opens=java.desktop/javax.swing=ALL-UNNAMED --add-opens=java.desktop/javax.swing.plaf.basic=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED --add-opens=java.desktop/sun.awt.X11=ALL-UNNAMED --add-opens=java.desktop/sun.awt.datatransfer=ALL-UNNAMED --add-opens=java.desktop/sun.awt.image=ALL-UNNAMED --add-opens=java.desktop/sun.awt=ALL-UNNAMED --add-opens=java.desktop/sun.font=ALL-UNNAMED --add-opens=java.desktop/sun.java2d=ALL-UNNAMED --add-opens=java.desktop/sun.swing=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing=ALL-UNNAMED --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED --add-opens=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED --add-opens=jdk.jdi/com.sun.tools.jdi=ALL-UNNAMED com.intellij.idea.Main
piotr     134635  2.1  5.9 667712 482672 ?       R    12:37   0:51 /usr/bin/Xvnc :1 -geometry 2560x1440 -auth /home/piotr/.Xauthority -desktop demon.local:1 (piotr) -fp catalogue:/etc/X11/fontpath.d -pn -rfbauth /home/piotr/.vnc/passwd -rfbport 5901
piotr     134870  1.5  5.2 3356416 425832 ?      Ssl  12:37   0:36 /usr/bin/plasmashell --no-respawn
piotr     134795  4.7  4.2 2068632 340448 ?      Ssl  12:37   1:53 /usr/bin/kwin_x11 --replace
piotr     135242  1.1  3.0 34753896 246960 ?     Ssl  12:37   0:28 /opt/google/chrome/chrome
piotr     136128  0.5  2.6 1212211500 209608 ?   Sl   12:37   0:12 /opt/google/chrome/chrome --type=renderer --crashpad-handler-pid=135332 --enable-crash-reporter=, --enable-dinosaur-easter-egg-alt-images --change-stack-guard-on-fork=enable --disable-gpu-compositing --lang=en-US --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=27 --time-ticks-at-unix-epoch=-1722939912823954 --launch-time-ticks=346360034043 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,11530877326889957697,6240751340682841738,262144 --variations-seed-version
piotr     136063  0.1  1.6 34381848 136436 ?     Sl   12:37   0:04 /opt/google/chrome/chrome --type=gpu-process --crashpad-handler-pid=135332 --enable-crash-reporter=, --change-stack-guard-on-fork=enable --gpu-preferences=WAAAAAAAAAAgAAAEAAAAAAAAAAAAAAAAAABgAAAAAAA4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAGAAAAAAAAAAYAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA== --use-gl=angle --use-angle=swiftshader-webgl --shared-files --field-trial-handle=3,i,11530877326889957697,6240751340682841738,262144 --variations-seed-version
piotr     139285 46.6  1.5 1587340 122644 ?      Sl   13:17   0:00 /usr/libexec/kscreenlocker_greet --graceTime 10000 --ksldfd 35
piotr     136267  0.0  1.4 1214079828 113092 ?   Sl   12:38   0:01 /opt/google/chrome/chrome --type=renderer --crashpad-handler-pid=135332 --enable-crash-reporter=, --enable-dinosaur-easter-egg-alt-images --change-stack-guard-on-fork=enable --disable-gpu-compositing --lang=en-US --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=34 --time-ticks-at-unix-epoch=-1722939912823954 --launch-time-ticks=346368758551 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,11530877326889957697,6240751340682841738,262144 --variations-seed-version
-----

sob, 10 sie 2024, 13:18:04 CEST
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
piotr     134635  2.9 74.0 10483096 5968240 ?    S    12:37   1:12 /usr/bin/Xvnc :1 -geometry 2560x1440 -auth /home/piotr/.Xauthority -desktop demon.local:1 (piotr) -fp catalogue:/etc/X11/fontpath.d -pn -rfbauth /home/piotr/.vnc/passwd -rfbport 5901
piotr     136778 10.8  1.1 8866448 95828 ?       Ssl  12:46   3:27 /home/piotr/apps/pycharm-2024.1.4/jbr/bin/java -classpath /home/piotr/apps/pycharm-2024.1.4/lib/platform-loader.jar:/home/piotr/apps/pycharm-2024.1.4/lib/util-8.jar:/home/piotr/apps/pycharm-2024.1.4/lib/util.jar:/home/piotr/apps/pycharm-2024.1.4/lib/app-client.jar:/home/piotr/apps/pycharm-2024.1.4/lib/util_rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/product.jar:/home/piotr/apps/pycharm-2024.1.4/lib/opentelemetry.jar:/home/piotr/apps/pycharm-2024.1.4/lib/app.jar:/home/piotr/apps/pycharm-2024.1.4/lib/product-client.jar:/home/piotr/apps/pycharm-2024.1.4/lib/modules.jar:/home/piotr/apps/pycharm-2024.1.4/lib/lib-client.jar:/home/piotr/apps/pycharm-2024.1.4/lib/stats.jar:/home/piotr/apps/pycharm-2024.1.4/lib/jps-model.jar:/home/piotr/apps/pycharm-2024.1.4/lib/external-system-rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/rd.jar:/home/piotr/apps/pycharm-2024.1.4/lib/bouncy-castle.jar:/home/piotr/apps/pycharm-2024.1.4/lib/protobuf.jar:/home/piotr/apps/pycharm-2024.1.4/lib/intellij-test-discovery.jar:/home/piotr/apps/pycharm-2024.1.4/lib/intellij-coverage-agent-1.0.744.jar:/home/piotr/apps/pycharm-2024.1.4/lib/forms_rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/lib.jar:/home/piotr/apps/pycharm-2024.1.4/lib/externalProcess-rt.jar:/home/piotr/apps/pycharm-2024.1.4/lib/groovy.jar:/home/piotr/apps/pycharm-2024.1.4/lib/annotations.jar:/home/piotr/apps/pycharm-2024.1.4/lib/async-profiler.jar:/home/piotr/apps/pycharm-2024.1.4/lib/grpc.jar:/home/piotr/apps/pycharm-2024.1.4/lib/jsch-agent.jar:/home/piotr/apps/pycharm-2024.1.4/lib/junit4.jar:/home/piotr/apps/pycharm-2024.1.4/lib/trove.jar -XX:ErrorFile=/home/piotr/java_error_in_pycharm_%p.log -XX:HeapDumpPath=/home/piotr/java_error_in_pycharm_.hprof -Xms256m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -XX:+IgnoreUnrecognizedVMOptions -ea -Dsun.io.useCanonCaches=false -Dsun.java2d.metal=true -Djbr.catch.SIGABRT=true -Djdk.http.auth.tunneling.disabledSchemes="" -Djdk.attach.allowAttachSelf=true -Djdk.module.illegalAccess.silent=true -Dkotlinx.coroutines.debug=off -XX:CICompilerCount=2 -XX:ReservedCodeCacheSize=512m -XX:CompileCommand=exclude,com/intellij/openapi/vfs/impl/FilePartNodeRoot,trieDescend -XX:SoftRefLRUPolicyMSPerMB=50 -Dide.show.tips.on.startup.default.value=false -Dsun.tools.attach.tmp.only=true -Dawt.lock.fair=true -Djava.net.preferIPv4Stack=true -Dawt.useSystemAAFontSettings=lcd -Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine -Dide.managed.by.toolbox=/home/piotr/.local/share/JetBrains/Toolbox/bin/jetbrains-toolbox -Dtoolbox.notification.token=8ae28aa5-1d2d-4c8b-b806-98fa259920d7 -Dtoolbox.notification.portFile=/home/piotr/.cache/JetBrains/Toolbox/ports/397276050.port -Dawt.toolkit.name=WLToolkit -Djb.vmOptionsFile=/home/piotr/.config/JetBrains/PyCharm2024.1/pycharm64.vmoptions -Djava.system.class.loader=com.intellij.util.lang.PathClassLoader -Didea.vendor.name=JetBrains -Didea.paths.selector=PyCharm2024.1 -Djna.boot.library.path=/home/piotr/apps/pycharm-2024.1.4/lib/jna/amd64 -Dpty4j.preferred.native.folder=/home/piotr/apps/pycharm-2024.1.4/lib/pty4j -Djna.nosys=true -Djna.noclasspath=true -Dintellij.platform.runtime.repository.path=/home/piotr/apps/pycharm-2024.1.4/modules/module-descriptors.jar -Didea.platform.prefix=Python -Dsplash=true -Daether.connector.resumeDownloads=false --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.ref=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/jdk.internal.vm=ALL-UNNAMED --add-opens=java.
base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.fs=ALL-UNNAMED --add-opens=java.base/sun.security.ssl=ALL-UNNAMED --add-opens=java.base/sun.security.util=ALL-UNNAMED --add-opens=java.base/sun.net.dns=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing.plaf.gtk=ALL-UNNAMED --add-opens=java.desktop/java.awt=ALL-UNNAMED --add-opens=java.desktop/java.awt.dnd.peer=ALL-UNNAMED --add-opens=java.desktop/java.awt.event=ALL-UNNAMED --add-opens=java.desktop/java.awt.image=ALL-UNNAMED --add-opens=java.desktop/java.awt.peer=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED --add-opens=java.desktop/javax.swing=ALL-UNNAMED --add-opens=java.desktop/javax.swing.plaf.basic=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED --add-opens=java.desktop/sun.awt.X11=ALL-UNNAMED --add-opens=java.desktop/sun.awt.datatransfer=ALL-UNNAMED --add-opens=java.desktop/sun.awt.image=ALL-UNNAMED --add-opens=java.desktop/sun.awt=ALL-UNNAMED --add-opens=java.desktop/sun.font=ALL-UNNAMED --add-opens=java.desktop/sun.java2d=ALL-UNNAMED --add-opens=java.desktop/sun.swing=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing=ALL-UNNAMED --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED --add-opens=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED --add-opens=jdk.jdi/com.sun.tools.jdi=ALL-UNNAMED com.intellij.idea.Main
piotr     139285 49.2  1.1 2663616 90640 ?       Dl   13:17   0:31 /usr/libexec/kscreenlocker_greet --graceTime 10000 --ksldfd 35
piotr     135242  1.2  0.9 34753896 73904 ?      Ssl  12:37   0:29 /opt/google/chrome/chrome
piotr     134795  5.4  0.6 2138392 54880 ?       Rsl  12:37   2:11 /usr/bin/kwin_x11 --replace
sddm       58838  0.8  0.6 1551620 53564 tty1    Sl+  sie07  36:01 /usr/bin/sddm-greeter-qt6 --socket /tmp/sddm--UfayHa
root       10946  0.3  0.5 3838760 45556 ?       Ssl  sie06  18:11 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
piotr     136128  0.5  0.3 1212211500 31560 ?    Sl   12:37   0:12 /opt/google/chrome/chrome --type=renderer --crashpad-handler-pid=135332 --enable-crash-reporter=, --enable-dinosaur-easter-egg-alt-images --change-stack-guard-on-fork=enable --disable-gpu-compositing --lang=en-US --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=27 --time-ticks-at-unix-epoch=-1722939912823954 --launch-time-ticks=346360034043 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,11530877326889957697,6240751340682841738,262144 --variations-seed-version
piotr     136293  0.1  0.3 1597328 26436 ?       Dsl  12:38   0:04 /usr/bin/konsole
-----

As seen, in the span of 1 minute memory usage of Xvnc went from 482672 kB (482 MB) to 5968240 kB (5.96GB) freezing the whole system having 8GB total memory.

CendioOssman commented 1 month ago

We probably need to run it via valgrind massif to figure out what's going on. If you could manage to set up a test with that, then it would be very helpful.

Please also make sure the TigerVNC debug packages are installed as we might not get good information otherwise.

piotr-dobrogost commented 1 month ago

The moment Xvnc allocated memory went from 376 588kB to 6 256 524kB:

wto, 13 sie 2024, 22:17:51 CEST
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
piotr     223283 11.8 27.3 8971996 2200176 ?     Ssl  21:42   4:08 /home/piotr/apps/pycharm-2024.2/jbr/bin/java -classpath /home/piotr/apps/pycharm-2024.2/lib/platform-loader.jar:/home/piotr/apps/pycharm-2024.2/lib/util-8.jar:/home/piotr/apps/pycharm-2024.2/lib/util.jar:/home/piotr/apps/pycharm-2024.2/lib/app-client.jar:/home/piotr/apps/pycharm-2024.2/lib/util_rt.jar:/home/piotr/apps/pycharm-2024.2/lib/product.jar:/home/piotr/apps/pycharm-2024.2/lib/opentelemetry.jar:/home/piotr/apps/pycharm-2024.2/lib/app.jar:/home/piotr/apps/pycharm-2024.2/lib/product-client.jar:/home/piotr/apps/pycharm-2024.2/lib/lib-client.jar:/home/piotr/apps/pycharm-2024.2/lib/stats.jar:/home/piotr/apps/pycharm-2024.2/lib/jps-model.jar:/home/piotr/apps/pycharm-2024.2/lib/external-system-rt.jar:/home/piotr/apps/pycharm-2024.2/lib/rd.jar:/home/piotr/apps/pycharm-2024.2/lib/bouncy-castle.jar:/home/piotr/apps/pycharm-2024.2/lib/protobuf.jar:/home/piotr/apps/pycharm-2024.2/lib/forms_rt.jar:/home/piotr/apps/pycharm-2024.2/lib/lib.jar:/home/piotr/apps/pycharm-2024.2/lib/externalProcess-rt.jar:/home/piotr/apps/pycharm-2024.2/lib/groovy.jar:/home/piotr/apps/pycharm-2024.2/lib/annotations.jar:/home/piotr/apps/pycharm-2024.2/lib/jsch-agent.jar:/home/piotr/apps/pycharm-2024.2/lib/junit4.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/trove.jar -XX:ErrorFile=/home/piotr/java_error_in_pycharm_%p.log -XX:HeapDumpPath=/home/piotr/java_error_in_pycharm_.hprof -Xms256m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -XX:+IgnoreUnrecognizedVMOptions -ea -Dsun.io.useCanonCaches=false -Dsun.java2d.metal=true -Djbr.catch.SIGABRT=true -Djdk.http.auth.tunneling.disabledSchemes="" -Djdk.attach.allowAttachSelf=true -Djdk.module.illegalAccess.silent=true -Dkotlinx.coroutines.debug=off -XX:CICompilerCount=2 -XX:ReservedCodeCacheSize=512m -XX:+UnlockDiagnosticVMOptions -XX:TieredOldPercentage=100000 -Dsun.tools.attach.tmp.only=true -Dawt.lock.fair=true -Djava.net.preferIPv4Stack=true -Dawt.useSystemAAFontSettings=lcd -Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine -Dide.managed.by.toolbox=/home/piotr/.local/share/JetBrains/Toolbox/bin/jetbrains-toolbox -Dtoolbox.notification.token=8ae28aa5-1d2d-4c8b-b806-98fa259920d7 -Dtoolbox.notification.portFile=/home/piotr/.cache/JetBrains/Toolbox/ports/397276050.port -Dawt.toolkit.name=XToolkit -Djb.vmOptionsFile=/home/piotr/.config/JetBrains/PyCharm2024.2/pycharm64.x11.vmoptions -Djava.system.class.loader=com.intellij.util.lang.PathClassLoader -Didea.vendor.name=JetBrains -Didea.paths.selector=PyCharm2024.2 -Djna.boot.library.path=/home/piotr/apps/pycharm-2024.2/lib/jna/amd64 -Dpty4j.preferred.native.folder=/home/piotr/apps/pycharm-2024.2/lib/pty4j -Djna.nosys=true -Djna.noclasspath=true -Dintellij.platform.runtime.repository.path=/home/piotr/apps/pycharm-2024.2/modules/module-descriptors.jar -Didea.platform.prefix=Python -Dsplash=true -Daether.connector.resumeDownloads=false --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.ref=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/jdk.internal.vm=ALL-UNNAMED --add-opens=java.base/sun.net.dns=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.fs=ALL-UNNAMED --add-opens=java.base/sun.security.ssl=ALL-UNNAMED --add-opens=java.base/sun.security.util=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing.plaf.gtk=ALL-UNNAMED --add-opens=java.desktop/java.awt=ALL-UNNAMED --add-opens=java.
desktop/java.awt.dnd.peer=ALL-UNNAMED --add-opens=java.desktop/java.awt.event=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED --add-opens=java.desktop/java.awt.image=ALL-UNNAMED --add-opens=java.desktop/java.awt.peer=ALL-UNNAMED --add-opens=java.desktop/javax.swing=ALL-UNNAMED --add-opens=java.desktop/javax.swing.plaf.basic=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED --add-opens=java.desktop/sun.awt=ALL-UNNAMED --add-opens=java.desktop/sun.awt.X11=ALL-UNNAMED --add-opens=java.desktop/sun.awt.datatransfer=ALL-UNNAMED --add-opens=java.desktop/sun.awt.image=ALL-UNNAMED --add-opens=java.desktop/sun.font=ALL-UNNAMED --add-opens=java.desktop/sun.java2d=ALL-UNNAMED --add-opens=java.desktop/sun.swing=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED --add-opens=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED --add-opens=jdk.jdi/com.sun.tools.jdi=ALL-UNNAMED -Xbootclasspath/a:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar com.intellij.idea.Main
piotr     221684  1.7  4.6 561032 376588 ?       S    21:42   0:36 /usr/bin/Xvnc :1 -geometry 2560x1440 -auth /home/piotr/.Xauthority -desktop demon.softax.local:1 (piotr) -fp catalogue:/etc/X11/fontpath.d -pn -rfbauth /home/piotr/.vnc/passwd -rfbport 5901
piotr     221917  0.6  4.4 3087132 359932 ?      Ssl  21:42   0:14 /usr/bin/plasmashell --no-respawn
piotr     222303  1.5  3.7 34748328 298580 ?     Ssl  21:42   0:32 /opt/google/chrome/chrome
piotr     223102  1.0  2.7 34462840 218280 ?     Sl   21:42   0:22 /opt/google/chrome/chrome --type=gpu-process --crashpad-handler-pid=222379 --enable-crash-reporter=, --change-stack-guard-on-fork=enable --gpu-preferences=WAAAAAAAAAAgAAAEAAAAAAAAAAAAAAAAAABgAAAAAAA4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAGAAAAAAAAAAYAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA== --use-gl=angle --use-angle=swiftshader-webgl --shared-files --field-trial-handle=3,i,13652203059422507086,3316981141428087593,262144 --variations-seed-version
piotr     223286  1.2  2.5 1212172240 206320 ?   Sl   21:42   0:27 /opt/google/chrome/chrome --type=renderer --crashpad-handler-pid=222379 --enable-crash-reporter=, --enable-dinosaur-easter-egg-alt-images --change-stack-guard-on-fork=enable --disable-gpu-compositing --lang=en-US --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=27 --time-ticks-at-unix-epoch=-1722939912823954 --launch-time-ticks=638253100761 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,13652203059422507086,3316981141428087593,262144 --variations-seed-version
piotr     221846  3.9  2.1 1914572 177016 ?      Ssl  21:42   1:25 /usr/bin/kwin_x11 --replace
piotr     223190  0.2  2.0 1212158036 167588 ?   Sl   21:42   0:06 /opt/google/chrome/chrome --type=renderer --crashpad-handler-pid=222379 --enable-crash-reporter=, --enable-dinosaur-easter-egg-alt-images --change-stack-guard-on-fork=enable --disable-gpu-compositing --lang=en-US --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=32 --time-ticks-at-unix-epoch=-1722939912823954 --launch-time-ticks=638250006462 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,13652203059422507086,3316981141428087593,262144 --variations-seed-version
piotr     223211  0.0  1.8 1214064668 152844 ?   Sl   21:42   0:01 /opt/google/chrome/chrome --type=renderer --crashpad-handler-pid=222379 --enable-crash-reporter=, --enable-dinosaur-easter-egg-alt-images --change-stack-guard-on-fork=enable --disable-gpu-compositing --lang=en-US --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=31 --time-ticks-at-unix-epoch=-1722939912823954 --launch-time-ticks=638251580985 --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,13652203059422507086,3316981141428087593,262144 --variations-seed-version
-----

wto, 13 sie 2024, 22:18:52 CEST
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
piotr     221684  2.4 77.6 9236520 6256524 ?     S    21:42   0:53 /usr/bin/Xvnc :1 -geometry 2560x1440 -auth /home/piotr/.Xauthority -desktop demon.softax.local:1 (piotr) -fp catalogue:/etc/X11/fontpath.d -pn -rfbauth /home/piotr/.vnc/passwd -rfbport 5901
piotr     222303  1.5  1.1 34752428 92004 ?      Ssl  21:42   0:34 /opt/google/chrome/chrome
piotr     226685 51.3  1.0 2663624 84324 ?       Sl   22:17   0:28 /usr/libexec/kscreenlocker_greet --graceTime 10000 --ksldfd 31
piotr     223283 11.7  0.9 8971996 78960 ?       Ssl  21:42   4:14 /home/piotr/apps/pycharm-2024.2/jbr/bin/java -classpath /home/piotr/apps/pycharm-2024.2/lib/platform-loader.jar:/home/piotr/apps/pycharm-2024.2/lib/util-8.jar:/home/piotr/apps/pycharm-2024.2/lib/util.jar:/home/piotr/apps/pycharm-2024.2/lib/app-client.jar:/home/piotr/apps/pycharm-2024.2/lib/util_rt.jar:/home/piotr/apps/pycharm-2024.2/lib/product.jar:/home/piotr/apps/pycharm-2024.2/lib/opentelemetry.jar:/home/piotr/apps/pycharm-2024.2/lib/app.jar:/home/piotr/apps/pycharm-2024.2/lib/product-client.jar:/home/piotr/apps/pycharm-2024.2/lib/lib-client.jar:/home/piotr/apps/pycharm-2024.2/lib/stats.jar:/home/piotr/apps/pycharm-2024.2/lib/jps-model.jar:/home/piotr/apps/pycharm-2024.2/lib/external-system-rt.jar:/home/piotr/apps/pycharm-2024.2/lib/rd.jar:/home/piotr/apps/pycharm-2024.2/lib/bouncy-castle.jar:/home/piotr/apps/pycharm-2024.2/lib/protobuf.jar:/home/piotr/apps/pycharm-2024.2/lib/forms_rt.jar:/home/piotr/apps/pycharm-2024.2/lib/lib.jar:/home/piotr/apps/pycharm-2024.2/lib/externalProcess-rt.jar:/home/piotr/apps/pycharm-2024.2/lib/groovy.jar:/home/piotr/apps/pycharm-2024.2/lib/annotations.jar:/home/piotr/apps/pycharm-2024.2/lib/jsch-agent.jar:/home/piotr/apps/pycharm-2024.2/lib/junit4.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/trove.jar -XX:ErrorFile=/home/piotr/java_error_in_pycharm_%p.log -XX:HeapDumpPath=/home/piotr/java_error_in_pycharm_.hprof -Xms256m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -XX:+IgnoreUnrecognizedVMOptions -ea -Dsun.io.useCanonCaches=false -Dsun.java2d.metal=true -Djbr.catch.SIGABRT=true -Djdk.http.auth.tunneling.disabledSchemes="" -Djdk.attach.allowAttachSelf=true -Djdk.module.illegalAccess.silent=true -Dkotlinx.coroutines.debug=off -XX:CICompilerCount=2 -XX:ReservedCodeCacheSize=512m -XX:+UnlockDiagnosticVMOptions -XX:TieredOldPercentage=100000 -Dsun.tools.attach.tmp.only=true -Dawt.lock.fair=true -Djava.net.preferIPv4Stack=true -Dawt.useSystemAAFontSettings=lcd -Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine -Dide.managed.by.toolbox=/home/piotr/.local/share/JetBrains/Toolbox/bin/jetbrains-toolbox -Dtoolbox.notification.token=8ae28aa5-1d2d-4c8b-b806-98fa259920d7 -Dtoolbox.notification.portFile=/home/piotr/.cache/JetBrains/Toolbox/ports/397276050.port -Dawt.toolkit.name=XToolkit -Djb.vmOptionsFile=/home/piotr/.config/JetBrains/PyCharm2024.2/pycharm64.x11.vmoptions -Djava.system.class.loader=com.intellij.util.lang.PathClassLoader -Didea.vendor.name=JetBrains -Didea.paths.selector=PyCharm2024.2 -Djna.boot.library.path=/home/piotr/apps/pycharm-2024.2/lib/jna/amd64 -Dpty4j.preferred.native.folder=/home/piotr/apps/pycharm-2024.2/lib/pty4j -Djna.nosys=true -Djna.noclasspath=true -Dintellij.platform.runtime.repository.path=/home/piotr/apps/pycharm-2024.2/modules/module-descriptors.jar -Didea.platform.prefix=Python -Dsplash=true -Daether.connector.resumeDownloads=false --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.ref=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/jdk.internal.vm=ALL-UNNAMED --add-opens=java.base/sun.net.dns=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.fs=ALL-UNNAMED --add-opens=java.base/sun.security.ssl=ALL-UNNAMED --add-opens=java.base/sun.security.util=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing=ALL-UNNAMED --add-opens=java.desktop/com.sun.java.swing.plaf.gtk=ALL-UNNAMED --add-opens=java.desktop/java.awt=ALL-UNNAMED --add-opens=java.
desktop/java.awt.dnd.peer=ALL-UNNAMED --add-opens=java.desktop/java.awt.event=ALL-UNNAMED --add-opens=java.desktop/java.awt.font=ALL-UNNAMED --add-opens=java.desktop/java.awt.image=ALL-UNNAMED --add-opens=java.desktop/java.awt.peer=ALL-UNNAMED --add-opens=java.desktop/javax.swing=ALL-UNNAMED --add-opens=java.desktop/javax.swing.plaf.basic=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text=ALL-UNNAMED --add-opens=java.desktop/javax.swing.text.html=ALL-UNNAMED --add-opens=java.desktop/sun.awt=ALL-UNNAMED --add-opens=java.desktop/sun.awt.X11=ALL-UNNAMED --add-opens=java.desktop/sun.awt.datatransfer=ALL-UNNAMED --add-opens=java.desktop/sun.awt.image=ALL-UNNAMED --add-opens=java.desktop/sun.font=ALL-UNNAMED --add-opens=java.desktop/sun.java2d=ALL-UNNAMED --add-opens=java.desktop/sun.swing=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED --add-opens=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED --add-opens=jdk.jdi/com.sun.tools.jdi=ALL-UNNAMED -Xbootclasspath/a:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar:/home/piotr/apps/pycharm-2024.2/lib/nio-fs.jar com.intellij.idea.Main
sddm      215929  0.8  0.6 1550724 53480 tty1    Sl+  15:56   3:07 /usr/bin/sddm-greeter-qt6 --socket /tmp/sddm--VsQirt
piotr     221846  4.5  0.6 1984844 53012 ?       Ssl  21:42   1:40 /usr/bin/kwin_x11 --replace
sddm      215968  0.0  0.4 1513168 32928 tty1    Sl+  15:56   0:00 /usr/bin/maliit-keyboard
piotr     222700  0.1  0.3 34035328 32124 ?      Sl   21:42   0:03 /opt/google/chrome/chrome --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=none --crashpad-handler-pid=222379 --enable-crash-reporter=, --change-stack-guard-on-fork=enable --shared-files=v8_context_snapshot_data:100 --field-trial-handle=3,i,13652203059422507086,3316981141428087593,262144 --variations-seed-version
sddm      215921  0.3  0.3 1075640 28056 tty1    Sl+  15:56   1:17 /usr/bin/kwin_wayland --no-global-shortcuts --no-lockscreen --inputmethod maliit-keyboard --locale1
-----

Output of journalctl -xeu vncserver@:1.service including time frame from the above:

sie 13 21:42:16 demon.local systemd[1]: Starting vncserver@:1.service - Remote desktop service (VNC)...
░░ Subject: A start job for unit vncserver@:1.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit vncserver@:1.service has begun execution.
░░
░░ The job identifier is 99303.
sie 13 21:42:16 demon.softax.local valgrind[221671]: ==221671== Massif, a heap profiler
sie 13 21:42:16 demon.softax.local valgrind[221671]: ==221671== Copyright (C) 2003-2024, and GNU GPL'd, by Nicholas Nethercote et al.
sie 13 21:42:16 demon.softax.local valgrind[221671]: ==221671== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
sie 13 21:42:16 demon.softax.local valgrind[221671]: ==221671== Command: /usr/libexec/vncsession-start :1
sie 13 21:42:16 demon.softax.local valgrind[221671]: ==221671==
sie 13 21:42:16 demon.softax.local valgrind[221672]: ==221672==
sie 13 21:42:16 demon.softax.local systemd[1]: Started vncserver@:1.service - Remote desktop service (VNC).
░░ Subject: A start job for unit vncserver@:1.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit vncserver@:1.service has finished successfully.
░░
░░ The job identifier is 99303.
sie 14 02:11:59 demon.local systemd[1]: vncserver@:1.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit vncserver@:1.service has successfully entered the 'dead' state.

Output from journalctl when oom-killer kills Xvnc: journal.txt

Output from valgrind massif generated using ms_print script – xvncmassive.txt

As far as I can see the output does not show big memory allocations, though… Reading the following from https://valgrind.org/docs/manual/ms-manual.html:

Peak snapshots are only ever taken after a deallocation happens. This avoids lots of unnecessary peak snapshot recordings (imagine what happens if your program allocates a lot of heap blocks in succession, hitting a new peak every time). But it means that if your program never deallocates any blocks, no peak will be recorded. It also means that if your program does deallocate blocks but later allocates to a higher peak without subsequently deallocating, the reported peak will be too low.

I guess the problem might be due to Xvnc being killed without dealocating memory which according to the above information would result in no peak being recorded?

Side note: I added memory limits to the /etc/systemd/system/vncserver@.service file:

[Service]
(…)
MemoryHigh=1G
MemoryMax=2G

to prevent Xvnc from allocating too much memory but it seems this have no effect and I don't know why.

CendioOssman commented 1 month ago

I'm afraid massif is attached to the wrong process. It's only tracking the small startup script, not Xvnc.

You could try letting it follow in to child process using --trace-children=no. You likely need to change the output locations as well to include %p to handle the different processes.

Alternatively, modify the vncserver script so that massif is injected in front of the call to Xvnc. Just make sure you don't get xinit instead.

piotr-dobrogost commented 1 month ago

I'm afraid massif is attached to the wrong process. It's only tracking the small startup script, not Xvnc.

Right. Service unit file has ExecStart=/usr/libexec/vncsession-start %i and /usr/libexec/vncsession-start is a bash script ending with exec "/usr/sbin/vncsession" "${USER}" "${DISPLAY}" As I did not use --trace-children valgrind stopped at this exec call.

You could try letting it follow in to child process using --trace-children=no. You likely need to change the output locations as well to include %p to handle the different processes.

Did you mean --trace-children=yes?

Alternatively, modify the vncserver script so that massif is injected in front of the call to Xvnc.

What do mean by vncserver script? It seems only /usr/libexec/vncsession-start script is used in my setup.

Just make sure you don't get xinit instead.

What do you mean? Looking at the process tree which I posted earlier vncsession(115666)───xinit(115667)─┬─Xvnc(115673) xinit is between vncsession which could be traced with massif and Xvnc which we are interested in so I see no way of omitting it.

I think injecting valgrind in the exec "/usr/sbin/vncsession" "${USER}" "${DISPLAY}" line and adding --trace-children=yes might work.

EDIT

When trying to run /usr/sbin/vncsession under valgrind the program quits:

[root@demon /]# /usr/bin/valgrind --trace-children=yes --tool=massif --massif-out-file=/var/log/vncmassif.%p /usr/sbin/vncsession piotr :1
==4641== Massif, a heap profiler
==4641== Copyright (C) 2003-2024, and GNU GPL'd, by Nicholas Nethercote et al.
==4641== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==4641== Command: /usr/sbin/vncsession piotr :1
==4641==
==4647==
==4669== could not unlink /tmp/vgdb-pipe-from-vgdb-to-4669-by-piotr-on-demon.local
==4669== could not unlink /tmp/vgdb-pipe-to-vgdb-from-4669-by-piotr-on-demon.local
==4669== could not unlink /tmp/vgdb-pipe-shared-mem-vgdb-4669-by-piotr-on-demon.local
==4641==
[root@demon /]# ==4648==

So I tried to run it without valgrind and it also quits. Journal shows:

sie 15 00:22:27 demon.local vncsession[4582]: pam_unix(tigervnc:session): session opened for user piotr(uid=1000) by piotr(uid=0)
sie 15 00:22:27 demon.local audit[4582]: USER_START pid=4582 uid=0 auid=1000 ses=8 subj=kernel msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_ke>
sie 15 00:22:27 demon.local vncsession[4582]: vncsession: vncserver exited with status=255
sie 15 00:22:27 demon.local vncsession[4582]: pam_unix(tigervnc:session): session closed for user piotr
sie 15 00:22:27 demon.local audit[4582]: USER_END pid=4582 uid=0 auid=1000 ses=8 subj=kernel msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_key>

How to run /usr/sbin/vncsession properly?

CendioOssman commented 1 month ago

Did you mean --trace-children=yes?

Yes, sorry. Brain fart.

What do mean by vncserver script? It seems only /usr/libexec/vncsession-start script is used in my setup.

It's executed by vncsession once a user session is created. It's what calls xinit.

Just make sure you don't get xinit instead.

What do you mean?

That was in case you wanted to avoid tracing everything, and having just Xvnc. But if we're going the everything route, then there is nothing to worry about. :)

How to run /usr/sbin/vncsession properly?

Run it from the service as you did before. It's not meant to be run by hand.

piotr-dobrogost commented 1 month ago

When trying to run /usr/sbin/vncsession under valgrind the program quits:

Well as I found out it quits but Xvnc process is running and I was able to connect via vnc to the host. As a result of running [root@demon /]# /usr/bin/valgrind --trace-children=yes --tool=massif --massif-out-file=/var/log/vncmassif.%p /usr/sbin/vncsession piotr :1 massif created 3 files:

[root@demon /]# ls /var/log/vncmassif.*
/var/log/vncmassif.6999  /var/log/vncmassif.7005  /var/log/vncmassif.7006

whereas Xvnc process had pid 5158 so no massif file for Xvnc was not created and I have no idea why. Also, the system after starting valgrind was extremely slow.

Run it from the service as you did before. It's not meant to be run by hand.

Unfortunately after adding --trace-children=yes option to valgrind call in the service unit file the service does not start:

sie 15 14:05:18 demon.local systemd[1]: Starting vncserver@:1.service - Remote desktop service (VNC)...
sie 15 14:05:18 demon.local valgrind[14150]: ==14150== Massif, a heap profiler
sie 15 14:05:18 demon.local valgrind[14150]: ==14150== Copyright (C) 2003-2024, and GNU GPL'd, by Nicholas Nethercote et al.
sie 15 14:05:18 demon.local valgrind[14150]: ==14150== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
sie 15 14:05:18 demon.local valgrind[14150]: ==14150== Command: /usr/libexec/vncsession-start :1
sie 15 14:05:18 demon.local valgrind[14150]: ==14150==
sie 15 14:05:18 demon.local valgrind[14154]: ==14154== Massif, a heap profiler
sie 15 14:05:18 demon.local valgrind[14154]: ==14154== Copyright (C) 2003-2024, and GNU GPL'd, by Nicholas Nethercote et al.
sie 15 14:05:18 demon.local valgrind[14154]: ==14154== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
sie 15 14:05:18 demon.local valgrind[14154]: ==14154== Command: /usr/bin/head -1
sie 15 14:05:18 demon.local valgrind[14154]: ==14154==
sie 15 14:05:18 demon.local valgrind[14155]: ==14155== Massif, a heap profiler
sie 15 14:05:18 demon.local valgrind[14155]: ==14155== Copyright (C) 2003-2024, and GNU GPL'd, by Nicholas Nethercote et al.
sie 15 14:05:18 demon.local valgrind[14155]: ==14155== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
sie 15 14:05:18 demon.local valgrind[14155]: ==14155== Command: /usr/bin/cut -d = -f 2-
sie 15 14:05:18 demon.local valgrind[14155]: ==14155==
sie 15 14:05:18 demon.local valgrind[14156]: ==14156== Massif, a heap profiler
sie 15 14:05:18 demon.local valgrind[14156]: ==14156== Copyright (C) 2003-2024, and GNU GPL'd, by Nicholas Nethercote et al.
sie 15 14:05:18 demon.local valgrind[14156]: ==14156== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
sie 15 14:05:18 demon.local valgrind[14156]: ==14156== Command: /usr/bin/sed s/\ *$//g
sie 15 14:05:18 demon.local valgrind[14156]: ==14156==
sie 15 14:05:19 demon.local valgrind[14154]: ==14154==
sie 15 14:05:19 demon.local valgrind[14155]: ==14155==
sie 15 14:05:19 demon.local valgrind[14156]: ==14156==
sie 15 14:05:19 demon.local valgrind[14152]: ==14152==
sie 15 14:05:19 demon.local valgrind[14150]: ==14150== Massif, a heap profiler
sie 15 14:05:19 demon.local valgrind[14150]: ==14150== Copyright (C) 2003-2024, and GNU GPL'd, by Nicholas Nethercote et al.
sie 15 14:05:19 demon.local valgrind[14150]: ==14150== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
sie 15 14:05:19 demon.local valgrind[14150]: ==14150== Command: /usr/sbin/vncsession piotr :1
sie 15 14:05:19 demon.local valgrind[14150]: ==14150==
sie 15 14:05:19 demon.local valgrind[14157]: ==14157==
sie 15 14:05:19 demon.local systemd-logind[1009]: New session 13 of user piotr.
sie 15 14:05:19 demon.local systemd[1]: Started session-13.scope - Session 13 of User piotr.
sie 15 14:05:19 demon.local vncsession[14158]: pam_unix(tigervnc:session): session opened for user piotr(uid=1000) by piotr(uid=0)
sie 15 14:05:19 demon.local audit[14158]: USER_START pid=14158 uid=0 auid=1000 ses=13 subj=kernel msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit
,pam_limits,pam_systemd,pam_unix acct="piotr" exe="/usr/sbin/vncsession" hostname=203.0.113.20 addr=203.0.113.20 terminal=? res=success'
sie 15 14:05:19 demon.local valgrind[14159]: ==14159== could not unlink /tmp/vgdb-pipe-from-vgdb-to-14159-by-piotr-on-???
sie 15 14:05:19 demon.local valgrind[14159]: ==14159== could not unlink /tmp/vgdb-pipe-to-vgdb-from-14159-by-piotr-on-???
sie 15 14:05:19 demon.local valgrind[14159]: ==14159== could not unlink /tmp/vgdb-pipe-shared-mem-vgdb-14159-by-piotr-on-???
sie 15 14:05:19 demon.local valgrind[14150]: ==14150==
sie 15 14:05:19 demon.local systemd[1]: Started vncserver@:1.service - Remote desktop service (VNC).
sie 15 14:05:19 demon.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=vncserver@:1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=?
terminal=? res=success'
sie 15 14:05:19 demon.local vncsession[14158]: vncsession: vncserver exited with status=1
sie 15 14:05:19 demon.local vncsession[14158]: pam_unix(tigervnc:session): session closed for user piotr
sie 15 14:05:19 demon.local audit[14158]: USER_END pid=14158 uid=0 auid=1000 ses=13 subj=kernel msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,
pam_limits,pam_systemd,pam_unix acct="piotr" exe="/usr/sbin/vncsession" hostname=203.0.113.20 addr=203.0.113.20 terminal=? res=success'
sie 15 14:05:19 demon.local valgrind[14158]: ==14158==
sie 15 14:05:20 demon.local systemd[1]: vncserver@:1.service: Deactivated successfully.

In light of above problems I think it's necessary to inject valgrind only for Xvnc process.

EDIT

On my system there are two perl scripts /usr/bin/vncserver and /usr/libexec/vncserver. The former ends with

sub NotifyAboutDeprecation
{
    warn "\nWARNING: vncserver has been replaced by a systemd unit and is now considered deprecated and removed in upstream.\n";
    warn "Please read /usr/share/doc/tigervnc/HOWTO.md for more information.\n";
}

so I guess it's no longer used so I focused on the latter one. I changed

# We build up our Xvnc command with options
push(@cmd, "/usr/bin/Xvnc", ":$displayNumber");

to

# We build up our Xvnc command with options
push(@cmd, "/usr/bin/valgrind", "--tool=massif", "--massif-out-file=/var/log/vncmassif.%p", "/usr/bin/Xvnc", ":$displayNumber");

but after this vncserver service does not start:

sie 15 15:15:13 demon.local systemd[1]: Starting vncserver@:1.service - Remote desktop service (VNC)...
sie 15 15:15:13 demon.local systemd-logind[927]: New session 9 of user piotr.
sie 15 15:15:13 demon.local systemd[1]: Started session-9.scope - Session 9 of User piotr.
sie 15 15:15:13 demon.local vncsession[3392]: pam_unix(tigervnc:session): session opened for user piotr(uid=1000) by piotr(uid=0)
sie 15 15:15:13 demon.local audit[3392]: USER_START pid=3392 uid=0 auid=1000 ses=9 subj=kernel msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="piotr" exe="/usr/sbin/vncsession" hostname=203.0.113.20 addr=203.0.113.20 terminal=? res=success'
sie 15 15:15:13 demon.local vncsession[3393]: ~/.vnc is deprecated, please consult 'man vncsession' for paths to migrate to.
sie 15 15:15:13 demon.local systemd[1]: Started vncserver@:1.service - Remote desktop service (VNC).
sie 15 15:15:13 demon.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=vncserver@:1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
sie 15 15:15:13 demon.local sudo[3367]: pam_unix(sudo:session): session closed for user root
sie 15 15:15:13 demon.local audit[3367]: USER_END pid=3367 uid=1000 auid=1000 ses=5 subj=kernel msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
sie 15 15:15:13 demon.local audit[3367]: CRED_DISP pid=3367 uid=1000 auid=1000 ses=5 subj=kernel msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
sie 15 15:15:13 demon.local vncsession[3392]: vncsession: vncserver exited with status=1
sie 15 15:15:13 demon.local vncsession[3392]: pam_unix(tigervnc:session): session closed for user piotr
sie 15 15:15:13 demon.local audit[3392]: USER_END pid=3392 uid=0 auid=1000 ses=9 subj=kernel msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="piotr" exe="/usr/sbin/vncsession" hostname=203.0.113.20 addr=203.0.113.20 terminal=? res=success'
sie 15 15:15:13 demon.local systemd[1]: vncserver@:1.service: Deactivated successfully.
sie 15 15:15:13 demon.local audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=vncserver@:1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
sie 15 15:15:13 demon.local systemd[1]: session-9.scope: Deactivated successfully.
sie 15 15:15:13 demon.local systemd-logind[927]: Session 9 logged out. Waiting for processes to exit.
sie 15 15:15:13 demon.local systemd-logind[927]: Removed session 9.
piotr-dobrogost commented 1 week ago

Any hints how to run Xvnc under valgrind's massif successfully?

CendioOssman commented 6 days ago

Your suggested modifications should work, as far as I can tell. Could you have a look in the user's VNC log file?