Xpra Server Crash with GLib >= 2.76

[weingo2]

Describe the bug I run xpra on Arch Linux on a desktop computer at home, so I can attach a thin client to it wherever I go. I start an xpra server with an xterm then open and close windows from that xterm (running tmux too). It may seem a bit archaic, but I primarily use cli tools that bring up plot windows. I also normally bind a websocket session to a port on the desktop machine then, forward that port over ssh to my local machine then attach to that port. This method has been easier for me than using the native xpra ssh.

Recently, I updated Arch Lunix on the desktop, and the xpra server started sporadically crashing when I open or close windows. When xpra crashes it seems to be segment faulting somewhere inside the GTK library and specifically inside glib. The session becomes "UNKNOWN" and I have to restart the session.

The stack trace from one of these crashes is here: gdb.txt A crash does not occur with every window open or close but seems to occur randomly.

To Reproduce Steps to reproduce the behavior:

1. xpra start --start=xterm --bind-ws=localhost:10001
2. Forward the port from the server to the client with ssh -L ...
3. xpra attach ws://localhost:10001
4. open and close windows from xterm

System Information (please complete the following information):

• Arch Linux
• Arch Linux or Windows 11
• 4.4.4
• 4.4.4

[totaam]

FYI: unless you also attach with a browser session and it is having problems with tcp socket upgrades to websockets, using bind-ws is counter productive and just a waste of bandwidth.

TIL:

#0  0x00007fd2cfb8089f in gdk_x11_display_translate_event (translator=<optimized out>, display=0x5590dd7e6380, event=0x5590dde1ab20, xevent=0x7ffe0ffec570) at ../gtk/gdk/x11/gdkdisplay-x11.c:653

GDK is choking on parsing of an X11 event.

You may be able to get more details by running the xpra server with -d all, though the log will grow very big, very quickly..

[weingo2]

Thank you for the information! I just figured websockets would be better because of the bidirectional channel.

So, I haven't been able to get it to crash in debug mode. I even started a server without debug mode again to check if the bug got resolved somehow, and it still crashed. I'm going to keep using my normal setup in debug mode to see if it ever crashes. I should have enough disk space to handle the log file. Any other things to try?

[totaam]

I just figured websockets would be better because of the bidirectional channel.

They all are - xpra cannot work without. https://github.com/Xpra-org/xpra/tree/master/docs/Network

Any other things to try?

Yes, the usual: disable every feature and every encoder (without debug mode) and see if it still crashes.

[weingo2]

server_tail.log

It finally died while in debug mode. I attached the last 1000 lines of the large log file that was produced. It should contain all the context of the error.

[totaam]

Looks like you pressed control-C in window 0x80000c:

2023-04-13 10:00:38,525 handle_key((1, True, 'c', 99, 54, ['control', 'mod2'], False, True))

Which caused a screen update:

2023-04-13 10:00:38,529 parse_event(..)=<X11:DamageNotify {'send_event': '0', 'serial': '0x35fe9', 'delivered_to': '0x80000c', 'window': '0x80000c', 'damage': '4194331', 'x': '34', 'y': '1282', 'width': '16', 'height': '32'}>

And window 0x600001 to close:

2023-04-13 10:00:38,539 parse_event(..)=<X11:UnmapNotify {'send_event': '0', 'serial': '0x35ff8', 'delivered_to': '0x600001', 'window': '0x600001'}>
...
2023-04-13 10:00:38,542 unmanaging window: WindowModel(0x600001) (<GdkX11.X11Window object at 0x7fe19c0fa480 (GdkX11Window at 0x55dc118b3680)> - <GdkX11.X11Window object at 0x7fe19c0fb080 (GdkX11Window at 0x55dc118b3920)>)

The window model cleanup code hits a bunch of errors we ignore since the window is already gone - I think that's safe:

xpra.gtk_common.error.XError: XError: BadDamage (invalid Damage parameter)

Then we get some more screen updates (DamageNotify on 0x80000c) - presumably some terminal window?

What does look a little bit suspicious is a bunch of cannot get gdk window for ..:

2023-04-13 10:00:38,562 cannot get gdk window for <GdkX11.X11Display object at 0x7fe1c769b880 (GdkX11Display at 0x55dc1119c7f0)>, 0x402007: constructor returned NULL

Perhaps b1d2e9eb81f1ac031e6ac95f34a82d284a2f8cda introduced a bug - in which case the commit above should fix that. Does that help? (without -d all to try to trigger it more readily)

[weingo2]

Often I just click control-c on my plotting to clear everything. However, it does crash sometimes when I close with window with the window X close button. I did only have one other window open at the time, an xterm window.

Your explanation of the log matches the observed behavior. When it crashes, the window I am trying to close closes, and the remaining windows flash before everything disappears. Naively I would think it still expects the window that just closed to exist and it seg faulting on a null pointer for that window. This doesn't explain why it only happens sometimes, though.

I just built xpra with that commit and installed it. It crashed using the same methodology...

What I have been trying to figure out is what changed about my setup. I've been using this xpra setup for a couple of years now with no issues then all the sudden, a month ago, this issue showed up. Is it a gtk issue? A gtk python binding issue? All these packages seem to have received small updates over the past month, so I couldn't narrow it down that way.

[totaam]

What I have been trying to figure out is what changed about my setup.

I believe that Arch must have upgraded an X11 or GTK library that makes it much more likely to hit the race condition that triggers this bug.

[weingo2]

Race condition makes sense. It's also a lot more reliable in debug mode, so the act of printing out a log message must slow down the right thread preventing the race condition. Any other debug suggestions? Honestly, I'm happily running in debug mode right now. I have plenty of disk space to handle the large log file it produces and don't notice any performance hits.

[totaam]

This may still trigger the bug and give us more details:

XPRA_X11_DEBUG_EVENTS="*" xpra start -d x11,gtk ...

[weingo2]

I ran this XPRA_X11_DEBUG_EVENTS="*" xpra start -d x11,gtk --start=xterm --bind-tcp=localhost:10001 and spawned and closed windows in a loop until it crashed. Here is the last 1000 lines of the log file it produced server_1000.log

[nentibusarchitectura]

I can trigger the same crash in Arch by opening and closing the SciTE editor, it happens all the time, can't tell for how long this has been the case, it happens with xpra 4.4.4-1 on my Arch system.

Simply opening SciTE and then File->Exit is enough to crash xpra on my machine.

[totaam]

Assuming that I find the time to install ArchLinux in a VM, are there any applications in the default repositories that can trigger this bug? Normally, this type of problem can be reproduced with gtkperf -a - does it reproduce it for you?

[weingo2]

I think any program that spawns windows has the potential to trigger this bug. I ran gtkperf -a from installed from the AUR (https://aur.archlinux.org/gtkperf.git) inside a shell script while loop until it crashed. It took a minute but eventually exhibited the bug's behavior.

Is there any other debug information I can provide you? If there is anything else you want to try, I would be happy to check out a development branch (or however you would want to distribute code) you create and test it.

[nentibusarchitectura]

Assuming that I find the time to install ArchLinux in a VM, are there any applications in the default repositories that can trigger this bug? Normally, this type of problem can be reproduced with gtkperf -a - does it reproduce it for you?

I can check this later too, I think (not hundred percent sure) that Firefox also triggered it for me, but SciTE was 100% consistent, it was to led me check if this issue had been reported.

[nentibusarchitectura]

Just did extensive testing xpra segfaults pretty much when closing or exiting any application, examples:

open xterm, type exit = crash open firefox go to file->quit = crash open firefox close window from x buton = crash open glxgears exit by pressing esc = crash etc...

Sometimes it doesn't crash, but only some times like 1 in 20.

I can provide a full core dump if required via private link.

xpra_crash_log.txt

At some point in the past it used to work fine in Arch, but since I do not do work in my Arch box often I have lost track of when and what version used to work.

It was by chance that I tested it a couple of days ago in Arch.

[thoefl]

Can confirm such segfaults occur on the current Gentoo testing (~amd64), but not on the stable branch. This might help narrowing down the version differences, e.g.: https://packages.gentoo.org/packages/dev-libs/glib

[totaam]

Gentoo might be easier as a testbed. I remember seeing an X11 commit recently that had to do with enabling some threading by default (could be xorg-server or libX11?), emerge-ing the older X11 ebuild might be enough to resolve this bug and tell us where the regression comes from. But it would be easier if we had a 100% reliable reproducer..

[thoefl]

libX11 and xorg-server are the same in stable and testing, but I just tried updating only glib (and its dependency gdbus-codegen) from 2.74.6 (stable) to 2.76.1 (testing), and that alone was enough to trigger segmentation faults in libgdk. I've attached the relevant dmesg output and the complete dependency trees of the broken and working configurations, but the only differences should be glib and gdbus-codegen.

This is using the older 4.3.4 release, but I had the same behaviour building from the current master branch.

dmesg.txt deps_segfault.txt deps_working.txt

EDIT: Possibly related, segfaults also reported in XFCE using glib 2.76: https://gitlab.gnome.org/GNOME/glib/-/issues/2941 https://gitlab.xfce.org/xfce/xfce4-session/-/issues/166

[totaam]

@nentibusarchitectura @weingo2 can you try downgrading glib to see if that fixes it?

I've scoured those gitlab links and there's nothing related to what we're doing here. The bugs seem to be related to gio and the file API whereas we're doing X11 window management..

For the record, the X11 threading change I was thinking about is this one: XInitThreads in library constructor breaks Motif! and I really don't think that this is relevant here: GTK does not call it and we never call X11 functions from other threads anyway, we even have thread checks in all the Cython bindings.

[nentibusarchitectura]

can you try downgrading glib to see if that fixes it?

My box uses glibc 2.37, I can only downgrade without screwing the system down to 2.36 which doesn't fix the issue, the crashes are still there.

Tried to downgrade xpra I can go as low as 4.3.3-r0 and the crashes are still there too.

Anything lower than 4.3.3 gives me python errors due to the system not meeting requirements due to rolling nature of dependencies in Arch.

Hope this helps.

[weingo2]

The arch changelog shows an upgrade from glib2 2.74.6-1 to glib2 2.76.1-2 over the last couple months.

I, too, tried to downgrade glib2 according to https://wiki.archlinux.org/title/Downgrading_packages by using the package cache method. That didn't work, and I quickly wound up with too many dependency conflicts. I then used the Arch Linux Archive method from https://wiki.archlinux.org/title/Arch_Linux_Archive to downgrade me to March 1st, 2023. This date has the older glib2-2.74.6-1 package in it. I also had to remove libgirepository with sudo pacman -Rd --nodeps libgirepository since it seems to have split out of another package and has conflicting files in it. This seemed to work but after rebooting my machine, Arch won't boot. The kernel won't load with efi for some reason... I am using a different distro with grub to boot arch, so that could be the problem.

Needless to say, I am out of commission until I solve the booting issues. If I don't get to it today, I won't be able to get to it until the Monday after next.

[weingo2]

So, I managed to recover my system and get the March 12th, 2023 Arch Linux Archive working. The day after this archive version, the glib library was upgraded. My system does not crash with this archive, I have been running gtkperf -a in a loop for 30 min with no crashes. This means that glib2 2.74.6-1 was stable on Arch for xpra until it was upgraded after March 12th. I can't say for sure that there is no other library causing problems but based on the Gentoo information from @thoefl I think it's safe to say that the glib library is causing the trouble.

[totaam]

I think it's safe to say that the glib library is causing the trouble.

I concur. I hope that versions between 2.74.x and 2.76.x are bisectable.

[borneoa]

Updating Arch installation with glib2 2.76.2-1 from "testing" repository does NOT solve the issue

[jpVm5jYYRE1VIKL]

Got same issue on Arch . before it was very old ubuntu switched to arch and now all broken cannot start any application. Constant crashes.

this is my startup command :

xpra start ssh:smth@10.10.10.254:10 --ssh=ssh --env=XPRA_FORCE_XSETINPUTFOCUS=1 --encoding=h264\
 --webcam=no --speaker=disabled --microphone=disabled --border=yellow --pulseaudio=no

Crash log :

(Xpra:13727): Gdk-CRITICAL **: 15:40:35.165: gdk_window_thaw_toplevel_updates: assertion 'window->update_and_descendants_freeze_count > 0' failed
(..)
  File "/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py", line 1689, in main
    return _Gtk_main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/xpra/x11/models/core.py", line 754, in do_xpra_client_message_event
    if not self.process_client_message_event(event):
  File "/usr/lib/python3.10/site-packages/xpra/x11/models/window.py", line 654, in process_client_message_event
    return super().process_client_message_event(event)
  File "/usr/lib/python3.10/site-packages/xpra/x11/models/base.py", line 611, in process_client_message_event
    log.info("Unhandled _NET_WM_STATE request: '%s'", event, atom1)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 461, in info
    self.log(logging.INFO, msg, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 452, in log
    global_logging_handler(self.logger.log, self.level_override or level, msg, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 142, in standard_logging
    log(level, msg, *args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1547, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/usr/lib/python3.10/site-packages/xpra/log.py", line 504, in emit
    self.handleError(record)
Message: "Unhandled _NET_WM_STATE request: '%s'"
Arguments: (<X11:ClientMessage {'send_event': '1', 'serial': '0x2993', 'delivered_to': '0x800004', 'window': '0x800004', 'message_type': '_NET_WM_STATE', 'format': '32', 'data': '(0, 432, 0, 1, 0)'}>, '_KDE_NET_WM_STATE_SKIP_SWITCHER')
2023-05-04 15:43:48,849  event<X11:ClientMessage {'send_event': '1', 'serial': '0x2993', 'delivered_to': '0x800004', 'window': '0x800004', 'message_type': '_NET_WM_STATE', 'format': '32', 'data': '(0, 432, 0, 1, 0)'}>
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:1689: Warning: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
  return _Gtk_main(*args, **kwargs)
2023-05-04 15:44:02,491 client   1 @36.728 window 3 has been moved to monitor 0: GSM DP-2 2560x1440 at 0,1440
2023-05-04 15:44:03,785 client   1 @38.020 Warning: window 3 requested static gravity
2023-05-04 15:44:03,785 client   1 @38.020  this is not implemented yet

Client show following :

2023-05-04 15:40:32,252 running, 1 window
2023-05-04 15:43:59,618 window 3 has been moved to monitor 0: GSM DP-2 2560x1440 at 0,1440
2023-05-04 15:44:00,909 Warning: window 3 requested static gravity
2023-05-04 15:44:00,910  this is not implemented yet
2023-05-04 15:44:16,021 Connection lost
2023-05-04 15:44:16,057 removing unix domain socket '/run/user/1000/xpra/clients/ktotoPC-97332'
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:1689: Warning: g_object_ref: assertion 'G_IS_OBJECT (object)' failed
  return _Gtk_main(*args, **kwargs)
CONNECTION_LOST, reconnecting
xpra initialization error:

[lotan]

I believe I suffer from the same problem. I'm also on Arch Linux, xpra version xpra v4.4.4-r0. Seemingly random crashes on all kinds of applications.

Error: 2405423 Segmentation fault (core dumped) XPRA_X11_DEBUG_EVENTS=* /usr/bin/xpra start -d all --no-daemon :11 The log shows:

       Stack trace of thread 2405423:
       #0  0x00007fb1cdc3a161 g_type_check_instance_is_a (libgobject-2.0.so.0 + 0x39161)
       #1  0x00007fb1ccbad690 n/a (libgdk-3.so.0 + 0x8b690)
       #2  0x00007fb1ccb53029 gdk_display_get_event (libgdk-3.so.0 + 0x31029)
       #3  0x00007fb1ccbada68 n/a (libgdk-3.so.0 + 0x8ba68)
       #4  0x00007fb1cdce053b g_main_context_dispatch (libglib-2.0.so.0 + 0x5a53b)
       #5  0x00007fb1cdd3d219 n/a (libglib-2.0.so.0 + 0xb7219)
       #6  0x00007fb1cdcdfc7f g_main_loop_run (libglib-2.0.so.0 + 0x59c7f)
       #7  0x00007fb1cbfd8e4f gtk_main (libgtk-3.so.0 + 0x1d8e4f)
       #8  0x00007fb1cdbfd4f6 n/a (libffi.so.8 + 0x74f6)
       #9  0x00007fb1cdbf9f5e n/a (libffi.so.8 + 0x3f5e)
       #10 0x00007fb1cdbfcb73 ffi_call (libffi.so.8 + 0x6b73)
       #11 0x00007fb1cddfc931 n/a (_gi.cpython-311-x86_64-linux-gnu.so + 0x2b931)
       #12 0x00007fb1cddfb2f0 n/a (_gi.cpython-311-x86_64-linux-gnu.so + 0x2a2f0)
       #13 0x00007fb1cf21f59d PyObject_Call (libpython3.11.so.1.0 + 0x21f59d)
       #14 0x00007fb1cf1f36bd _PyEval_EvalFrameDefault (libpython3.11.so.1.0 + 0x1f36bd)
       #15 0x00007fb1cf2a9f6a n/a (libpython3.11.so.1.0 + 0x2a9f6a)
       #16 0x00007fb1cf2a997c PyEval_EvalCode (libpython3.11.so.1.0 + 0x2a997c)
       #17 0x00007fb1cf2c86b3 n/a (libpython3.11.so.1.0 + 0x2c86b3)
       #18 0x00007fb1cf2c43ba n/a (libpython3.11.so.1.0 + 0x2c43ba)
       #19 0x00007fb1cf2dadd3 n/a (libpython3.11.so.1.0 + 0x2dadd3)
       #20 0x00007fb1cf2d9ef4 _PyRun_SimpleFileObject (libpython3.11.so.1.0 + 0x2d9ef4)
       #21 0x00007fb1cf2d8de8 _PyRun_AnyFileObject (libpython3.11.so.1.0 + 0x2d8de8)
       #22 0x00007fb1cf2d3722 Py_RunMain (libpython3.11.so.1.0 + 0x2d3722)
       #23 0x00007fb1cf29bf6b Py_BytesMain (libpython3.11.so.1.0 + 0x29bf6b)
       #24 0x00007fb1cee39850 n/a (libc.so.6 + 0x23850)
       #25 0x00007fb1cee3990a __libc_start_main (libc.so.6 + 0x2390a)
       #26 0x00005620af367045 _start (python3.11 + 0x1045)

Of course I'm happy to provide a core dump.

It does look like a problem in gtk.

Unfortunately, I can't downgrade far.

[thesamesam]

I think it's safe to say that the glib library is causing the trouble.

I concur. I hope that versions between 2.74.x and 2.76.x are bisectable.

glib-2.76 removed its "slice allocator" which covered up memory management bugs. You can force the "new" allocator in older glib for testing with: G_SLICE=always-malloc xpra .... Running it under Valgrind is likely to show issues as well.

See https://gitlab.gnome.org/GNOME/glib/-/issues/2941 and https://gitlab.gnome.org/GNOME/glib/-/issues/2937.

[totaam]

@thesamesam that helps, I can reproduce the crash with 2.74 which I have installed - the problem is that python and valgrind don't get on very well, and the bug could still be somewhere in GTK or the gi bindings.

[frej]

I think I have the same problem on Gentoo with glib 2.76.2, at least the back-traces when segfaulting look similar.

If I run valgrind --trace-children=yes --trace-children-skip='/usr/bin/xterm,/usr/bin/Xvfb' --log-file=/tmp/vg/%p.log /usr/bin/python3 /usr/bin/xpra start :20 --exit-with-children=yes --no-daemon --start-child=xterm, then patching the client code to avoid a timeout (as valgrind makes is slow) with:

diff --git a/xpra/client/base/client_base.py b/xpra/client/base/client_base.py
index ea090b2a3..8b1f425e7 100644
--- a/xpra/client/base/client_base.py
+++ b/xpra/client/base/client_base.py
@@ -358,7 +358,7 @@ class XpraClientBase(ServerInfoMixin, FilePrintMixin):
                                         DEFAULT_KEY_HASH, DEFAULT_KEYSIZE, DEFAULT_ITERATIONS, INITIAL_PADDING)
         self.have_more = protocol.source_has_more
         if conn.timeout>0:
-            self.timeout_add((conn.timeout + EXTRA_TIMEOUT) * 1000, self.verify_connected)
+            self.timeout_add((conn.timeout + EXTRA_TIMEOUT) * 1000 * 3000, self.verify_connected)
         process = getattr(conn, "process", None)        #ie: ssh is handled by another process
         if process:
             proc, name, command = process

Then do xpra attach :20 and when the xterm pops up (it takes 45s+) I run while true ; do xterm -e false ; done in it. When that has run for a while I check the Valgrind log file for the pid for the server (check for the printout running with pid <pid> on .... In it, I find a bunch of uninitialized reads from libx265, but what is interesting is this:

==29866== Invalid read of size 8
==29866==    at 0x944F473: gdk_xid_equal (gdkxid.c:41)
==29866==    by 0x7A37018: g_hash_table_lookup_node (ghash.c:495)
==29866==    by 0x7A37018: g_hash_table_lookup (ghash.c:1541)
==29866==    by 0x944F648: gdk_x11_window_lookup_for_display (gdkxid.c:105)
==29866==    by 0x9439023: gdk_event_source_get_filter_window (gdkeventsource.c:120)
==29866==    by 0x9439023: gdk_event_source_translate_event (gdkeventsource.c:189)
==29866==    by 0x9439023: _gdk_x11_display_queue_events (gdkeventsource.c:341)
==29866==    by 0x94043CF: gdk_display_get_event (gdkdisplay.c:442)
==29866==    by 0x9438E11: gdk_event_source_dispatch (gdkeventsource.c:363)
==29866==    by 0x7A4B31B: g_main_dispatch (gmain.c:3460)
==29866==    by 0x7A4B31B: g_main_context_dispatch (gmain.c:4200)
==29866==    by 0x7A4B5B7: g_main_context_iterate.constprop.0 (gmain.c:4276)
==29866==    by 0x7A4B86E: g_main_loop_run (gmain.c:4479)
==29866==    by 0x99F861C: gtk_main (gtkmain.c:1329)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==  Address 0xc5594f0 is 32 bytes inside a block of size 96 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x9420536: gdk_window_finalize (gdkwindow.c:581)
==29866==    by 0x7B884EA: g_object_unref (gobject.c:3938)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x944DBAD: _gdk_x11_display_create_window_impl (gdkwindow-x11.c:1063)
==29866==    by 0x9419D2F: gdk_window_new (gdkwindow.c:1512)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866== 
==29866== Invalid read of size 8
==29866==    at 0x9439038: gdk_event_source_get_filter_window (gdkeventsource.c:123)
==29866==    by 0x9439038: gdk_event_source_translate_event (gdkeventsource.c:189)
==29866==    by 0x9439038: _gdk_x11_display_queue_events (gdkeventsource.c:341)
==29866==    by 0x94043CF: gdk_display_get_event (gdkdisplay.c:442)
==29866==    by 0x9438E11: gdk_event_source_dispatch (gdkeventsource.c:363)
==29866==    by 0x7A4B31B: g_main_dispatch (gmain.c:3460)
==29866==    by 0x7A4B31B: g_main_context_dispatch (gmain.c:4200)
==29866==    by 0x7A4B5B7: g_main_context_iterate.constprop.0 (gmain.c:4276)
==29866==    by 0x7A4B86E: g_main_loop_run (gmain.c:4479)
==29866==    by 0x99F861C: gtk_main (gtkmain.c:1329)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD1BA: pygi_function_cache_invoke (pygi-cache.c:862)
==29866==  Address 0xc5cb970 is 0 bytes inside a block of size 408 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==    by 0x79B205E: pygobject_emit (pygobject-object.c:1932)
==29866==    by 0x49DD611: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x9419A94: gdk_window_new (gdkwindow.c:1392)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD042: _function_cache_invoke_real (pygi-cache.c:783)
==29866==    by 0x79CD042: _constructor_cache_invoke_real (pygi-cache.c:929)
==29866== 
==29866== Invalid read of size 8
==29866==    at 0x7BA6735: g_type_check_instance_is_a (gtype.c:4146)
==29866==    by 0x943904C: gdk_event_source_get_filter_window (gdkeventsource.c:123)
==29866==    by 0x943904C: gdk_event_source_translate_event (gdkeventsource.c:189)
==29866==    by 0x943904C: _gdk_x11_display_queue_events (gdkeventsource.c:341)
==29866==    by 0x94043CF: gdk_display_get_event (gdkdisplay.c:442)
==29866==    by 0x9438E11: gdk_event_source_dispatch (gdkeventsource.c:363)
==29866==    by 0x7A4B31B: g_main_dispatch (gmain.c:3460)
==29866==    by 0x7A4B31B: g_main_context_dispatch (gmain.c:4200)
==29866==    by 0x7A4B5B7: g_main_context_iterate.constprop.0 (gmain.c:4276)
==29866==    by 0x7A4B86E: g_main_loop_run (gmain.c:4479)
==29866==    by 0x99F861C: gtk_main (gtkmain.c:1329)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==  Address 0xc5cb970 is 0 bytes inside a block of size 408 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==    by 0x79B205E: pygobject_emit (pygobject-object.c:1932)
==29866==    by 0x49DD611: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x9419A94: gdk_window_new (gdkwindow.c:1392)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD042: _function_cache_invoke_real (pygi-cache.c:783)
==29866==    by 0x79CD042: _constructor_cache_invoke_real (pygi-cache.c:929)
==29866== 
==29866== Invalid read of size 8
==29866==    at 0x944F473: gdk_xid_equal (gdkxid.c:41)
==29866==    by 0x7A37018: g_hash_table_lookup_node (ghash.c:495)
==29866==    by 0x7A37018: g_hash_table_lookup (ghash.c:1541)
==29866==    by 0x944F648: gdk_x11_window_lookup_for_display (gdkxid.c:105)
==29866==    by 0x944C52B: gdk_x11_window_foreign_new_for_display (gdkwindow-x11.c:1244)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD042: _function_cache_invoke_real (pygi-cache.c:783)
==29866==    by 0x79CD042: _constructor_cache_invoke_real (pygi-cache.c:929)
==29866==    by 0x79CD1BA: pygi_function_cache_invoke (pygi-cache.c:862)
==29866==    by 0x79BCCE9: _callable_info_call (pygi-info.c:548)
==29866==    by 0xA5204F1: ??? (in /usr/lib/python3.11/site-packages/xpra/x11/gtk3/gdk_bindings.cpython-311-x86_64-linux-gnu.so)
==29866==  Address 0xc5594f0 is 32 bytes inside a block of size 96 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x9420536: gdk_window_finalize (gdkwindow.c:581)
==29866==    by 0x7B884EA: g_object_unref (gobject.c:3938)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x944DBAD: _gdk_x11_display_create_window_impl (gdkwindow-x11.c:1063)
==29866==    by 0x9419D2F: gdk_window_new (gdkwindow.c:1512)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866== 
==29866== Invalid read of size 8
==29866==    at 0x7BA6C05: g_type_check_instance_is_fundamentally_a (gtype.c:4164)
==29866==    by 0x7B8813D: g_object_ref (gobject.c:3775)
==29866==    by 0x944C538: gdk_x11_window_foreign_new_for_display (gdkwindow-x11.c:1245)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD042: _function_cache_invoke_real (pygi-cache.c:783)
==29866==    by 0x79CD042: _constructor_cache_invoke_real (pygi-cache.c:929)
==29866==    by 0x79CD1BA: pygi_function_cache_invoke (pygi-cache.c:862)
==29866==    by 0x79BCCE9: _callable_info_call (pygi-info.c:548)
==29866==    by 0xA5204F1: ??? (in /usr/lib/python3.11/site-packages/xpra/x11/gtk3/gdk_bindings.cpython-311-x86_64-linux-gnu.so)
==29866==    by 0xA527B5B: ??? (in /usr/lib/python3.11/site-packages/xpra/x11/gtk3/gdk_bindings.cpython-311-x86_64-linux-gnu.so)
==29866==  Address 0xc5cb970 is 0 bytes inside a block of size 408 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==    by 0x79B205E: pygobject_emit (pygobject-object.c:1932)
==29866==    by 0x49DD611: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x9419A94: gdk_window_new (gdkwindow.c:1392)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD042: _function_cache_invoke_real (pygi-cache.c:783)
==29866==    by 0x79CD042: _constructor_cache_invoke_real (pygi-cache.c:929)
==29866== 
==29866== Invalid read of size 8
==29866==    at 0x944F473: gdk_xid_equal (gdkxid.c:41)
==29866==    by 0x7A37018: g_hash_table_lookup_node (ghash.c:495)
==29866==    by 0x7A37018: g_hash_table_lookup (ghash.c:1541)
==29866==    by 0x944F648: gdk_x11_window_lookup_for_display (gdkxid.c:105)
==29866==    by 0x943323E: gdk_x11_display_translate_event (gdkdisplay-x11.c:647)
==29866==    by 0x9439711: _gdk_x11_event_translator_translate (gdkeventtranslator.c:51)
==29866==    by 0x9439222: gdk_event_source_translate_event (gdkeventsource.c:243)
==29866==    by 0x9439222: _gdk_x11_display_queue_events (gdkeventsource.c:341)
==29866==    by 0x94043CF: gdk_display_get_event (gdkdisplay.c:442)
==29866==    by 0x9438E11: gdk_event_source_dispatch (gdkeventsource.c:363)
==29866==    by 0x7A4B31B: g_main_dispatch (gmain.c:3460)
==29866==    by 0x7A4B31B: g_main_context_dispatch (gmain.c:4200)
==29866==    by 0x7A4B5B7: g_main_context_iterate.constprop.0 (gmain.c:4276)
==29866==    by 0x7A4B86E: g_main_loop_run (gmain.c:4479)
==29866==    by 0x99F861C: gtk_main (gtkmain.c:1329)
==29866==  Address 0xc5594f0 is 32 bytes inside a block of size 96 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x9420536: gdk_window_finalize (gdkwindow.c:581)
==29866==    by 0x7B884EA: g_object_unref (gobject.c:3938)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x944DBAD: _gdk_x11_display_create_window_impl (gdkwindow-x11.c:1063)
==29866==    by 0x9419D2F: gdk_window_new (gdkwindow.c:1512)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866== 
==29866== Invalid read of size 8
==29866==    at 0x9433253: gdk_x11_display_translate_event (gdkdisplay-x11.c:653)
==29866==    by 0x9439711: _gdk_x11_event_translator_translate (gdkeventtranslator.c:51)
==29866==    by 0x9439222: gdk_event_source_translate_event (gdkeventsource.c:243)
==29866==    by 0x9439222: _gdk_x11_display_queue_events (gdkeventsource.c:341)
==29866==    by 0x94043CF: gdk_display_get_event (gdkdisplay.c:442)
==29866==    by 0x9438E11: gdk_event_source_dispatch (gdkeventsource.c:363)
==29866==    by 0x7A4B31B: g_main_dispatch (gmain.c:3460)
==29866==    by 0x7A4B31B: g_main_context_dispatch (gmain.c:4200)
==29866==    by 0x7A4B5B7: g_main_context_iterate.constprop.0 (gmain.c:4276)
==29866==    by 0x7A4B86E: g_main_loop_run (gmain.c:4479)
==29866==    by 0x99F861C: gtk_main (gtkmain.c:1329)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==  Address 0xc5cb970 is 0 bytes inside a block of size 408 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==    by 0x79B205E: pygobject_emit (pygobject-object.c:1932)
==29866==    by 0x49DD611: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x9419A94: gdk_window_new (gdkwindow.c:1392)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD042: _function_cache_invoke_real (pygi-cache.c:783)
==29866==    by 0x79CD042: _constructor_cache_invoke_real (pygi-cache.c:929)
==29866== 
==29866== Invalid read of size 8
==29866==    at 0x7BA6735: g_type_check_instance_is_a (gtype.c:4146)
==29866==    by 0x9433267: gdk_x11_display_translate_event (gdkdisplay-x11.c:653)
==29866==    by 0x9439711: _gdk_x11_event_translator_translate (gdkeventtranslator.c:51)
==29866==    by 0x9439222: gdk_event_source_translate_event (gdkeventsource.c:243)
==29866==    by 0x9439222: _gdk_x11_display_queue_events (gdkeventsource.c:341)
==29866==    by 0x94043CF: gdk_display_get_event (gdkdisplay.c:442)
==29866==    by 0x9438E11: gdk_event_source_dispatch (gdkeventsource.c:363)
==29866==    by 0x7A4B31B: g_main_dispatch (gmain.c:3460)
==29866==    by 0x7A4B31B: g_main_context_dispatch (gmain.c:4200)
==29866==    by 0x7A4B5B7: g_main_context_iterate.constprop.0 (gmain.c:4276)
==29866==    by 0x7A4B86E: g_main_loop_run (gmain.c:4479)
==29866==    by 0x99F861C: gtk_main (gtkmain.c:1329)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==  Address 0xc5cb970 is 0 bytes inside a block of size 408 free'd
==29866==    at 0x484300B: free (vg_replace_malloc.c:974)
==29866==    by 0x7BA572B: g_type_free_instance (gtype.c:2062)
==29866==    by 0x79B1271: pygobject_clear (pygobject-object.c:1234)
==29866==    by 0x79B1271: pygobject_dealloc (pygobject-object.c:1129)
==29866==    by 0x4A290DC: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4ABE6FF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA438F: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B83450: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==    by 0x79B205E: pygobject_emit (pygobject-object.c:1932)
==29866==    by 0x49DD611: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==  Block was alloc'd at
==29866==    at 0x48456FF: calloc (vg_replace_malloc.c:1554)
==29866==    by 0x7A51F70: g_malloc0 (gmem.c:163)
==29866==    by 0x7BA5223: g_type_create_instance (gtype.c:1965)
==29866==    by 0x7B8903F: g_object_new_internal (gobject.c:2246)
==29866==    by 0x7B8A79B: g_object_new_with_properties (gobject.c:2409)
==29866==    by 0x7B8B3D8: g_object_new (gobject.c:2055)
==29866==    by 0x9419A94: gdk_window_new (gdkwindow.c:1392)
==29866==    by 0x7BD4FF9: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4493: ??? (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x7BD4B8C: ffi_call (in /usr/lib64/libffi.so.8.1.2)
==29866==    by 0x79CB437: pygi_invoke_c_callable (pygi-invoke.c:684)
==29866==    by 0x79CD042: _function_cache_invoke_real (pygi-cache.c:783)
==29866==    by 0x79CD042: _constructor_cache_invoke_real (pygi-cache.c:929)
==29866== 
==29866== Conditional jump or move depends on uninitialised value(s)
==29866==    at 0x49FA987: PyLong_FromLong (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79CB809: _invoke_marshal_out_args (pygi-invoke.c:640)
==29866==    by 0x79CB809: pygi_invoke_c_callable (pygi-invoke.c:711)
==29866==    by 0x79CD1BA: pygi_function_cache_invoke (pygi-cache.c:862)
==29866==    by 0x79BCCE9: _callable_info_call (pygi-info.c:548)
==29866==    by 0x49D184A: _PyObject_MakeTpCall (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4985585: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x4AA43BF: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x49D4661: ??? (in /usr/lib64/libpython3.11.so.1.0)
==29866==    by 0x79C6A7A: pyg_signal_class_closure_marshal (pygi-type.c:1047)
==29866==    by 0x7B8350F: g_closure_invoke (gclosure.c:832)
==29866==    by 0x7B96538: signal_emit_unlocked_R.isra.0 (gsignal.c:3851)
==29866==    by 0x79B205E: pygobject_emit (pygobject-object.c:1932)

So a use-after-free looks likely.

[totaam]

Thanks @frej !

In it, I find a bunch of uninitialized reads from libx265

x265 should not be enabled, please complain to your package maintainer.

while true ; do xterm -e false ; done

Might be easier with gtkperf -a.

but what is interesting is this:

It looks like the event handlers end up trying to find a window created through gdk_window_new but already deallocated via pygobject_dealloc.

Running xpra with -d x11, I see this just before the crash:

2023-05-13 22:36:01,717 x_event_filter event=('xpra-destroy-event', None)/DestroyNotify window=0x56e
2023-05-13 22:36:01,717 parse_event(..)=<X11:DestroyNotify {'send_event': '0', 'serial': '3815', 'delivered_to': '0x56e', 'window': '0xa00001'}>
2023-05-13 22:36:01,717 x_event_filter event=('xpra-destroy-event', None)/DestroyNotify took 0.1ms
2023-05-13 22:36:01,717 x_event_filter event=('xpra-unmap-event', 'xpra-child-unmap-event')/UnmapNotify window=0xa00003
2023-05-13 22:36:01,717 parse_event(..)=<X11:UnmapNotify {'send_event': '0', 'serial': '3815', 'delivered_to': '0xa00003', 'window': '0xa00003'}>
2023-05-13 22:36:01,717 do_xpra_unmap_event(<X11:UnmapNotify {'send_event': '0', 'serial': '3815', 'delivered_to': '0xa00003', 'window': '0xa00003'}>) client window unmapped, last_unmap_serial=0x0
2023-05-13 22:36:01,717 unmanaging window: WindowModel(a00003) (<GdkX11.X11Window object at 0x7fe84d832140 (GdkX11Window at 0x562aa9edda90)> - <GdkX11.X11Window object at 0x7fe863e4b3c0 (GdkX11Window at 0x562aaa5bbdd0)>)
2023-05-13 22:36:01,718 XError detected while already in unwind; discarding
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 125, in safe_x_exit
    self.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadDrawable (invalid Pixmap or Window parameter)
2023-05-13 22:36:01,718 XError detected while already in unwind; discarding
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 125, in safe_x_exit
    self.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadWindow (invalid Window parameter)
2023-05-13 22:36:01,718 couldn't TranslateCoordinates (maybe window is gone)
2023-05-13 22:36:01,718 XError detected while already in unwind; discarding
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 125, in safe_x_exit
    self.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadWindow (invalid Window parameter)
2023-05-13 22:36:01,718 Window.do_unmanaged(False) damage_forward_handle=220, composite=CompositeHelper(a00003)
2023-05-13 22:36:01,718 close_damage_handle()
2023-05-13 22:36:01,719 invalidating named pixmap, contents handle=PixmapWrapper(0x40008e, 550, 300)
2023-05-13 22:36:01,719 PixmapWrapper(0x40008e, 550, 300).cleanup()
2023-05-13 22:36:01,719 XError detected while already in unwind; discarding
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 125, in safe_x_exit
    self.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadDamage (invalid Damage parameter)
2023-05-13 22:36:01,719 XShmWrapper.cleanup() ref_count=5
2023-05-13 22:36:01,719 invalidating named pixmap, contents handle=None
2023-05-13 22:36:01,719 XError detected while already in unwind; discarding
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 125, in safe_x_exit
    self.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadWindow (invalid Window parameter)
2023-05-13 22:36:01,719 scrub_x11() x11 properties=['WM_STATE', '_NET_FRAME_EXTENTS', '_NET_WM_ALLOWED_ACTIONS']
2023-05-13 22:36:01,719 XError detected while already in unwind; discarding
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 125, in safe_x_exit
    self.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadWindow (invalid Window parameter)
2023-05-13 22:36:01,720 x_event_filter event=('xpra-unmap-event', 'xpra-child-unmap-event')/UnmapNotify took 2.7ms
2023-05-13 22:36:01,720 x_event_filter event=('xpra-focus-out-event', None)/FocusOut window=0xa00003
2023-05-13 22:36:01,720 parse_event(..)=<X11:FocusOut {'send_event': '0', 'serial': '3815', 'delivered_to': '0xa00003', 'window': '0xa00003', 'mode': '0', 'detail': '1'}>
2023-05-13 22:36:01,720 x_event_filter event=('xpra-focus-out-event', None)/FocusOut took 0.2ms

[chewi]

x265 should not be enabled, please complain to your package maintainer.

Sorted. In my defence, it doesn't say that anywhere.

[totaam]

Sorted. In my defence, it doesn't say that anywhere.

All the codecs are automatically enabled when present, this one is not and you have to force enable it explicitly. That's the hint you were looking for.

[totaam]

FWIW, I've tried running with XPRA_X_SYNC=1 XPRA_SYNCHRONIZE=1 and that makes no difference. Another data point is that the crash only occurs when non-override-redirect windows are unmapped. OR windows are more lightweight, but they still share a lot of common code: XShmImageWrapper, UnmapNotify, etc

[frej]

@totaam, I've been trying to figure out why we get a read from freed memory. To me it looks like a bug in Gtk/gdk. This is what I think happens (follow along in the source code for gtk-3.24.37:

1) A GtkWindow is destroyed and then unreffed by the python GC. Eventually we end up in gdk_window_finalize() which unrefs window->impl which triggers gdk_window_impl_x11_finalize().

2) gdk_window_impl_x11_finalize() in turn sees that the window is destroyed and does not call _gdk_x11_display_remove_window: https://github.com/GNOME/gtk/blob/013b629a2e5420cc01647f859226eb1a37ebe661/gdk/x11/gdkwindow-x11.c#L529-L536

3) The destroy event eventually trickles through to gdk_event_source_get_filter_window() which calls gdk_x11_window_lookup_for_display, but as the object registered in the hash table has been de-allocated in step 1 we segfault (this is what we see in the Valgrind log).

If we could still read the freed memory as with glib 2.74 and the slice allocator, we would have looked up the window and triggered gdk_window_destroy_notify() on it which would call _gdk_x11_display_remove_window() and all would be well.

[edit:] \<pure-speculation>Are we racing the destroy notification in the xpra code? I haven't looked at the source code, but is it possible that we decide to destroy a window, then drop the reference on the python side. If the destroy event goes through Gtk before the GC unrefs the reference all is well, if the GC happens first we crash?\</pure-speculation>

[totaam]

@frej I have edited your comment to include permalinks to the actual GTK source code instead of filenames + line numbers.

How did you figure out that:

• .. _trigger gdk_window_impl_x11_finalize() ?
• .. _eventually trickles through to [gdk_event_source_get_filter_window()]_?

is it possible that we decide to destroy a window, then drop the reference on the python side

We don't actually "destroy" the gdk window in the python code. We only respond to DestroyNotify and UnmapNotify events telling us that the window is already gone, so that we can forward those events to the client(s):

• these X11 events are mapped to python names here: https://github.com/Xpra-org/xpra/blob/d98af749bd0b88a0657a6055e39d56455b428901/xpra/x11/gtk3/gdk_bindings.pyx#L550-L551
• and sent to a bunch of listeners to clean things up:
  • for regular windows
  • override-redirect windows
  • for both: invalidate their pixmap You can see the details of these events being processed by xpra using: XPRA_X11_DEBUG_EVENTS="DestroyNotify,UnmapNotify" xpra start ...

---

Eventually, once everything has been cleaned up in xpra, the Gdk window's refcount should be zero. It may not happen exactly when we receive the event (in that one function call) because we use a signal to notify listeners, and AFAIK signals are sent through the main loop - which may have other pending events in it. I don't think this should be an issue though, because everything is asynchronous already.

I also assume that GDK_WINDOW_DESTROYED is called automatically when we use a gdk window through the gi bindings.

---

Now, the one part that I was starting to look into is this one: https://github.com/Xpra-org/xpra/blob/d98af749bd0b88a0657a6055e39d56455b428901/xpra/x11/gtk3/gdk_bindings.pyx#L805-L835 It converts X11 window xid into a gdk window objects so that the event objects that those handlers receive have gdk window objects in them rather than xids. I don't expect the call to GdkX11.X11Window.foreign_new_for_display to somehow mismanage the refcount (though I have seen the cannot get gdk window for ... message in the logs when things go wrong - could just be harmless), but perhaps the surrounding ugly code that deals with gdk_x11_display_error_trap_push / gdk_x11_display_error_trap_pop can somehow prevent Gdk from seeing the X11 UnmapNotify event when it should?

[frej]

How did you figure out that:

.. _trigger gdk_window_impl_x11_finalize() ?

It's in the Valgrind report for the first invalid read, it tells you from where the block we're accessing was freed. Combined with: inserted printfs in Gtk to print out the xid and address of the window object when windows are created and finalized; printfs on entry and exit to the functions adding and removing mappings in the table consulted by gdk_x11_window_lookup_for_display; And some printfs before each call to _gdk_x11_display_remove_window in the Gtk, you can track the pointers and figure out the sequence of events which caused to the invalid read.

.. eventually trickles through to [gdk_event_source_get_filter_window()]?

This is also from the Valgrind report (the call stack) combined with the printf in gdk_x11_window_lookup_for_display.

I have little time to work on this during the work week, but I'll take a new look in the weekend.

We don't actually "destroy" the gdk window in the python code.

But you drop the reference to it (lets it get out of scope), which is what triggers the free, and as GDK_WINDOW_DESTROYED() returns true, _gdk_x11_display_remove_window was not called in the finalizer. To me it looks like the Gdk never expected that to happen (Gdk assumes that GDK_WINDOW_DESTROYED()`) should be false if the window is destroyed by dropping the last reference). When does xpra forget a window, on unmap or destroy?

[totaam]

I started replacing all GTK window wrappers with raw X11 xid (see #1995 for details - worthy goal anyway) and eventually found the culprit Gdk.Window.destroy: https://github.com/Xpra-org/xpra/blob/ae69e0479fae581207756e69d1c42ed97e2b0f4d/xpra/x11/models/window.py#L389 I'm not sure what to replace it with though: https://lazka.github.io/pgi-docs/Gdk-3.0/classes/Window.html#Gdk.Window.destroy says: Destroys the window system resources associated with self and decrements self's reference count. The window system resources for all children of self are also destroyed, but the children’s reference counts are not decremented. More importantly: Note that a window will not be destroyed automatically when its reference count reaches zero. You must call this function yourself before that happens. So we have to destroy it, somehow.

[totaam]

@frej the commit above fixes things by not using GTK for the corral window, but there are problems with it:

• it is going to be hard to backport to older branches
• it is incomplete, ie: the get_server_time needs implementing
• it may introduce other bugs due to the differences between GTK and plain X11 calls

All these issues are solvable, but I would much rather find a way to make GTK not crash when calling destroy, any suggestions?

[neuhalje]

Ubuntu lunar uses glibc 2.67 and also constantly crashes XPra when closing a window.

[totaam]

@neuhalje I don't think that the version of glibc, aka libc-bin, is relevant. This ticket is about GLib, aka libglib and Ubuntu Lunar ships version 2.76.1 which is problematic.

[neuhalje]

@totaam I wasn't sure. But the names, versions, and symptoms were too close to ignore it ,-)

[frej]

All these issues are solvable, but I would much rather find a way to make GTK not crash when calling destroy, any suggestions?

Sorry @totaam, I don't know enough about the Xpra internals to suggest anything intelligent. To me, using raw Xlib seems a much saner option than relying on Gdk which I think is mainly intended for Gtk's own use. Perhaps you can use a GtkWindow instead, in the hope that Gtk will manage the life cycle of the GdkWindow properly? The backing GdkWindow is retrievable using gtk_widget_get_window for when you need access to the low level stuff.

As for a replacement for get_server_time, as far as I can see, you are only using the server time to set the focus. In that case, why not use Xlib's XSetInputFocus with CurrentTime, now that your are no longer using Gtk?

[totaam]

@frej GtkWindow: interesting idea, I will give it a go.

As for the the focus stuff, we already use both XSetInputFocus and WM_TAKE_FOCUS as per the ewmh spec: https://github.com/Xpra-org/xpra/blob/5e9b18b26f72a10c5c810c62161974fd939eef1b/xpra/x11/models/window.py#L797-L826 The comment there reads: Have to fetch the time, not just use CurrentTime Which is OK now that I have re-implemented the function: 8004b42fc3bab36aebb8fd4919babbce5b26a770

[frej]

The comment there reads: Have to fetch the time, not just use CurrentTime

I see, bad @frej!

[totaam]

So, I tried the GTK approach, which had the merit of being simple to implement but I didn't get very far with it because it caused too many intractable problems with focus and other events going MIA. The minimal fix above is somewhat bigger than most stable branch fixes, but most of it is boilerplate Cython headers cut&pasted, the actual change to the window model code is minimal: https://github.com/Xpra-org/xpra/commit/a0cc7229a0ee8443e6b99d587e94143905bad337#diff-a4160a51eb3e2d697991cbebafcd5941baa16ee796c86f6282271fb1e3a41091

@neuhalje @frej @chewi @thesamesam @weingo2 please give this commit a try, if I don't hear anything in the next 48 hours then 4.4.5 will be released with this fix as its headline.

[frej]

Can confirm that e2dfae92f2811f9c3fe0f9a33e9274d11ebaff94 fixes the segfault I experienced with glib 2.76.2.

[frej]

Unfortunately the fix in e2dfae92f2811f9c3fe0f9a33e9274d11ebaff94, although it fixes the the segfault experienced with glib 2.76.2, appears to prevent Firefox from creating its initial window.

/usr/bin/python3 /usr/bin/xpra start :20 --exit-with-children=yes --no-daemon --start-child=xterm followed by xpra attach gives me an xterm. Starting new xterms and exiting them works fine (it previously segfaulted xpra). Starting Firefox from the initial xterm does not produce a new Firefox window. No errors are shown, either from the two instances of xpra nor in the xterm. pstree on the xterm shows the normal set of firefox threads and processes.

This is with 4.4.4 [edit: corrected version] with e2dfae92f2811f9c3fe0f9a33e9274d11ebaff94 cherry picked, as well as with the complete v4.4.x branch at 4a4f45d5e39c814d495203b9cd0910cffc846a06.

[totaam]

Saw this with XPRA_LOG_MANAGE_FAILURES=1:

2023-05-22 11:07:47,324 failed to setup 0xa0003d
Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/xpra/x11/models/core.py", line 311, in call_setup
    self.setup()
  File "/usr/lib64/python3.11/site-packages/xpra/x11/models/window.py", line 192, in setup
    prop_set(self.corral_xid, "_NET_WM_NAME", "utf8", "Xpra-CorralWindow-%#x" % self.xid)
  File "/usr/lib64/python3.11/site-packages/xpra/x11/gtk_x11/prop.py", line 102, in prop_set
    raw_prop_set(xid, key, dtype, dformat, data)
  File "/usr/lib64/python3.11/site-packages/xpra/x11/gtk_x11/prop.py", line 107, in raw_prop_set
    with xsync:
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 196, in __exit__
    trap.Xexit()
  File "/usr/lib64/python3.11/site-packages/xpra/gtk_common/error.py", line 121, in Xexit
    raise XError(error)
xpra.gtk_common.error.XError: XError: BadWindow (invalid Window parameter)
2023-05-22 11:07:47,325 cannot manage Window 0xa0003d: XError: BadWindow (invalid Window parameter)
(..)
xpra.x11.common.Unmanageable: XError: BadWindow (invalid Window parameter)
2023-05-22 11:07:47,327  window name: Mozilla Firefox
2023-05-22 11:07:47,327  window info: unmapped window Mozilla Firefox (0, 0, 1282, 1026)

That's because Firefox (wrongly - but that's a different issue) uses a 32-bit window with transparency and these are hard to create with the correct arguments. We were trying to copy the window attributes from the client window, now we just get most of them from the root window instead. (24-bit depth and we inherit the visual)

[weingo2]

The seg fault crash seems to be fixed but I also cannot open firefox, thunderbird, or google-chrome (from the aur). Basic guis I run like tcl/tk programs or runelite seem to be fine.

Running gimp closed the xpra connection but the server was still in the LIVE state, requiring me to reconnect. But once I reconnected, it worked fine.

So, I'd say this specific bug is fixed by https://github.com/Xpra-org/xpra/commit/a0cc7229a0ee8443e6b99d587e94143905bad337#diff-a4160a51eb3e2d697991cbebafcd5941baa16ee796c86f6282271fb1e3a41091

Are the firefox, thunderbird, chrome issues worth a separate bug or is it just something you will be aware of in the future?

@totaam Thank you for the bug fix!

[totaam]

Are the firefox, thunderbird, chrome issues worth a separate bug or is it just something you will be aware of in the future?

@weingo2 you need c7ddee5a5c959d0aa4ecf612dfc447b5febd6b5d as per https://github.com/Xpra-org/xpra/issues/3822#issuecomment-1556548207