search

chimera-linux / cports

Chimera ports collection
BSD 2-Clause "Simplified" License
173 stars 110 forks source link

ppc64le: Crash in terminal emulators based on VTE, NULL pointer in `gnutls_cipher_set_iv()` #2469

Closed AlbertoGP closed 1 month ago

AlbertoGP commented 1 month ago

I have no idea why the terminal would need TLS.

This happens consistently when any command prints enough output, for instance: ls -R /usr It does not need to be that much, the example ls crashes in less than one second. When re-launching it from lldb without first exiting the debugger, it crashes immediately before it outputs anything.

This crashes all running instances of the terminal emulator, so what I do is to start lldb in one, in this case console (Gnome terminal), and launch another one, in this case xfce4-terminal.

I have not tested on other CPUs. This has been happening for a few days, and persisted over several apk upgrades.

$ lldb /usr/bin/xfce4-terminal 
(lldb) target create "/usr/bin/xfce4-terminal"
Current executable set to '/usr/bin/xfce4-terminal' (powerpc64le).
(lldb) run
Process 16872 launched: '/usr/bin/xfce4-terminal' (powerpc64le)
Process 16872 stopped
* thread #1, name = 'xfce4-terminal', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) bt
* thread #1, name = 'xfce4-terminal', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00003fffed92ed38 libgnutls.so.30`gnutls_cipher_set_iv + 72
    frame #2: 0x00003fffef8c6bb0 libvte-2.91.so.0`_vte_boa_write(_VteBoa*, unsigned long, char const*) + 576
    frame #3: 0x00003fffef8c5210 libvte-2.91.so.0`_vte_file_stream_append(_VteStream*, char const*, unsigned long) + 224
    frame #4: 0x00003fffef8c4730 libvte-2.91.so.0`_vte_stream_append + 48
    frame #5: 0x00003fffef870ca4 libvte-2.91.so.0`vte::base::Ring::freeze_row(unsigned long, _VteRowData const*) + 1380
    frame #6: 0x00003fffef872634 libvte-2.91.so.0`vte::base::Ring::insert(unsigned long, unsigned char) + 580
    frame #7: 0x00003fffef899a54 libvte-2.91.so.0`vte::terminal::Terminal::ring_insert(long, bool) + 148
    frame #8: 0x00003fffef87f374 libvte-2.91.so.0`vte::terminal::Terminal::scroll_text_up(vte::scrolling_region const&, long, bool) + 372
    frame #9: 0x00003fffef8bd590 libvte-2.91.so.0`vte::terminal::Terminal::LF(vte::parser::Sequence const&) + 128
    frame #10: 0x00003fffef883e9c libvte-2.91.so.0`vte::terminal::Terminal::process_incoming_utf8(vte::terminal::Terminal::ProcessingContext&, vte::base::Chunk&) + 1660
    frame #11: 0x00003fffef88295c libvte-2.91.so.0`vte::terminal::Terminal::process_incoming() + 444
    frame #12: 0x00003fffef897cb0 libvte-2.91.so.0`vte::terminal::Terminal::process() + 176
    frame #13: 0x00003fffef8991a0 libvte-2.91.so.0`vte::terminal::process_timeout(_GtkWidget*, void*) + 64
    frame #14: 0x00003fffef875204 libvte-2.91.so.0`scheduler_tick_callback(_GtkWidget*, _GdkFrameClock*, void*) + 100
    frame #15: 0x00003fffeeedc314 libgtk-3.so.0`gtk_widget_on_frame_clock_update.llvm.401478336712267111 + 180
    frame #16: 0x00003fffee1b02bc libgobject-2.0.so.0`g_cclosure_marshal_VOID__VOIDv + 76
    frame #17: 0x00003fffee1ab378 libgobject-2.0.so.0`___lldb_unnamed_symbol924 + 472
    frame #18: 0x00003fffee1d5d30 libgobject-2.0.so.0`___lldb_unnamed_symbol1136 + 1840
    frame #19: 0x00003fffee1d67f8 libgobject-2.0.so.0`g_signal_emit + 136
    frame #20: 0x00003fffee52e8d4 libgdk-3.so.0`gdk_frame_clock_paint_idle + 788
    frame #21: 0x00003fffee5098a8 libgdk-3.so.0`gdk_threads_dispatch.llvm.16695620749143848822 + 120
    frame #22: 0x00003fffedd200ac libglib-2.0.so.0`___lldb_unnamed_symbol2521 + 172
    frame #23: 0x00003fffedd2ba5c libglib-2.0.so.0`___lldb_unnamed_symbol2540 + 540
    frame #24: 0x00003fffedd2c260 libglib-2.0.so.0`___lldb_unnamed_symbol2541 + 896
    frame #25: 0x00003fffedd2cb6c libglib-2.0.so.0`g_main_loop_run + 476
    frame #26: 0x00003fffeec739a8 libgtk-3.so.0`gtk_main + 136
    frame #27: 0x000000010002be9c xfce4-terminal`main + 1084
    frame #28: 0x00003fffefa66008 libc.so
(lldb)
q66 commented 1 month ago

probably af6477db5bb024306452d47ccb52081adb96898a then?

nekopsykose commented 1 month ago

I have no idea why the terminal would need TLS.

gnutls is a general purpose crypto library so they use some methods, it has nothing to do with tls

nekopsykose commented 1 month ago

hm i can't reproduce this so it might be ppc specific..

nekopsykose commented 1 month ago

could you install the debug symbols and try to print the locals at frame 1 and whatnot?

AlbertoGP commented 1 month ago

could you install the debug symbols and try to print the locals at frame 1 and whatnot?

If you give me the exact sequence of steps needed for that I’ll be happy to do so.

nekopsykose commented 1 month ago

apk add musl-dbg xfce4-terminal-dbg vte-common-dbg vte-gtk3-dbg gtk+3-dbg glib-dbg gnutls-dbg

in the debugger, f 1, fr v

AlbertoGP commented 1 month ago

I’m on it.

q66 commented 1 month ago

you could also try downgrading the version and see what happens

nekopsykose commented 1 month ago

and you can also try stick a options = ["!lto"] and rebuild it

AlbertoGP commented 1 month ago

I had to add the main-debug and main-contrib-debug repositories:

apk add chimera-repo-main-debug
apk add chimera-repo-contrib-debug # For xfce4-terminal-dbg
apk add musl-dbg xfce4-terminal-dbg vte-common-dbg vte-gtk3-dbg gtk+3-dbg glib-dbg gnutls-dbg
$ lldb /usr/bin/xfce4-terminal 
(lldb) target create "/usr/bin/xfce4-terminal"
Current executable set to '/usr/bin/xfce4-terminal' (powerpc64le).
(lldb) run
Process 9543 launched: '/usr/bin/xfce4-terminal' (powerpc64le)
Process 9543 stopped
* thread #1, name = 'xfce4-terminal', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) bt
* thread #1, name = 'xfce4-terminal', stop reason = signal SIGSEGV: address not mapped to object (fault address: 0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00003fffed92ed38 libgnutls.so.30`gnutls_cipher_set_iv [inlined] _gnutls_cipher_setiv(handle=0x00003fffecc0b0d0, iv=0x0000000000000000, ivlen=70368421260848) at cipher_int.h:77:9
    frame #2: 0x00003fffed92ed10 libgnutls.so.30`gnutls_cipher_set_iv(handle=0x00003fffecc0b0d0, iv=0x00003fffecc0aed8, ivlen=12) at crypto-api.c:188:6
    frame #3: 0x00003fffef8c6bb0 libvte-2.91.so.0`_vte_boa_write(_VteBoa*, unsigned long, char const*) [inlined] _vte_boa_encrypt(boa=0x00003fffecc0ae30, offset=0, overwrite_counter=<unavailable>, data="\U00000013", len=<unavailable>) at vtestream-file.h:722:9
    frame #4: 0x00003fffef8c6ba4 libvte-2.91.so.0`_vte_boa_write(boa=0x00003fffecc0ae30, offset=0, data="") at vtestream-file.h:1012:9
    frame #5: 0x00003fffef8c5210 libvte-2.91.so.0`_vte_file_stream_append(astream=0x00003fffed0163f0, data=<unavailable>, len=<unavailable>) at vtestream-file.h:1194:25
    frame #6: 0x00003fffef8c4730 libvte-2.91.so.0`_vte_stream_append(stream=<unavailable>, data=<unavailable>, len=<unavailable>) at vtestream-base.h:76:2
    frame #7: 0x00003fffef870900 libvte-2.91.so.0`vte::base::Ring::freeze_row(this=0x00003fffea9141f8, position=<unavailable>, row=0x00003fffea511710) at ring.cc:335:5
    frame #8: 0x00003fffef872634 libvte-2.91.so.0`vte::base::Ring::insert(unsigned long, unsigned char) [inlined] vte::base::Ring::freeze_one_row(this=0x00003fffea9141f8) at ring.cc:746:2
    frame #9: 0x00003fffef872610 libvte-2.91.so.0`vte::base::Ring::insert(unsigned long, unsigned char) [inlined] vte::base::Ring::maybe_freeze_one_row(this=0x00003fffea9141f8) at ring.cc:796:3
    frame #10: 0x00003fffef8725f4 libvte-2.91.so.0`vte::base::Ring::insert(this=0x00003fffea9141f8, position=<unavailable>, bidi_flags='\x01') at ring.cc:929:2
    frame #11: 0x00003fffef899a54 libvte-2.91.so.0`vte::terminal::Terminal::ring_insert(this=0x00003fffea914060, position=<unavailable>, fill=<unavailable>) at vteinternal.hh:822:29
    frame #12: 0x00003fffef87f374 libvte-2.91.so.0`vte::terminal::Terminal::scroll_text_up(vte::scrolling_region const&, long, bool) [inlined] vte::terminal::Terminal::ring_append(this=0x00003fffea914060, fill=<unavailable>) at vteinternal.hh:829:24
    frame #13: 0x00003fffef87f364 libvte-2.91.so.0`vte::terminal::Terminal::scroll_text_up(this=0x00003fffea914060, scrolling_region=<unavailable>, amount=0, fill=<unavailable>) at vte.cc:2770:25
    frame #14: 0x00003fffef8bd590 libvte-2.91.so.0`vte::terminal::Terminal::LF(vte::parser::Sequence const&) [inlined] vte::terminal::Terminal::line_feed(this=<unavailable>) at vteseq.cc:1119:9
    frame #15: 0x00003fffef8bd510 libvte-2.91.so.0`vte::terminal::Terminal::LF(this=<unavailable>, seq=<unavailable>) at vteseq.cc:6267:9
    frame #16: 0x00003fffef883e9c libvte-2.91.so.0`vte::terminal::Terminal::process_incoming_utf8(this=0x00003fffea914060, context=0x00003fffffffea60, chunk=0x00003fffe8806640) at parser-cmd-handlers.hh:224:1
    frame #17: 0x00003fffef88295c libvte-2.91.so.0`vte::terminal::Terminal::process_incoming(this=0x00003fffea914060) at vte.cc:3896:25
    frame #18: 0x00003fffef897cb0 libvte-2.91.so.0`vte::terminal::Terminal::process() [inlined] vte::terminal::Terminal::time_process_incoming(this=0x00003fffea914060) at vte.cc:10883:2
    frame #19: 0x00003fffef897c94 libvte-2.91.so.0`vte::terminal::Terminal::process(this=0x00003fffea914060) at vte.cc:10903:25
    frame #20: 0x00003fffef8991a0 libvte-2.91.so.0`vte::terminal::process_timeout(widget=<unavailable>, data=0x00003fffea914060) at vte.cc:10922:32
    frame #21: 0x00003fffef875204 libvte-2.91.so.0`scheduler_tick_callback(widget=0x00003fffef211ad0, frame_clock=<unavailable>, user_data=0x00003fffecd38a00) at scheduler.cc:118:9
    frame #22: 0x00003fffeeedc314 libgtk-3.so.0`gtk_widget_on_frame_clock_update(frame_clock=0x00003fffe9f0f300, widget=0x00003fffef211ad0) at gtkwidget.c:5287:15
    frame #23: 0x00003fffee1b02bc libgobject-2.0.so.0`g_cclosure_marshal_VOID__VOIDv(closure=<unavailable>, return_value=<unavailable>, instance=<unavailable>, args=<unavailable>, marshal_data=<unavailable>, n_params=<unavailable>, param_types=<unavailable>) at gmarshal.c:165:3
    frame #24: 0x00003fffee1ab378 libgobject-2.0.so.0`_g_closure_invoke_va(closure=0x00003fffed031410, return_value=<unavailable>, instance=<unavailable>, args=<unavailable>, n_params=0, param_types=0x0000000000000000) at gclosure.c:897:7
    frame #25: 0x00003fffee1d5d30 libgobject-2.0.so.0`signal_emit_valist_unlocked(instance=0x00003fffe9f0f300, signal_id=42, detail=<unavailable>, var_args="") at gsignal.c:3424:8
    frame #26: 0x00003fffee1d67f8 libgobject-2.0.so.0`g_signal_emit [inlined] g_signal_emit_valist(instance=0x00003fffe9f0f300, signal_id=42, detail=0, var_args="") at gsignal.c:3263:7
    frame #27: 0x00003fffee1d67d8 libgobject-2.0.so.0`g_signal_emit(instance=<unavailable>, signal_id=<unavailable>, detail=<unavailable>) at gsignal.c:3583:3
    frame #28: 0x00003fffee52e8d4 libgdk-3.so.0`gdk_frame_clock_paint_idle [inlined] _gdk_frame_clock_emit_update(frame_clock=0x00003fffe9f0f300) at gdkframeclock.c:645:3
    frame #29: 0x00003fffee52e8c4 libgdk-3.so.0`gdk_frame_clock_paint_idle(data=0x00003fffe9f0f300) at gdkframeclockidle.c:547:19
    frame #30: 0x00003fffee5098a8 libgdk-3.so.0`gdk_threads_dispatch(data=0x00003fffe80718c0) at gdk.c:769:11
    frame #31: 0x00003fffedd200ac libglib-2.0.so.0`g_timeout_dispatch(source=<unavailable>, callback=<unavailable>, user_data=<unavailable>) at gmain.c:4989:15
    frame #32: 0x00003fffedd2ba5c libglib-2.0.so.0`g_main_context_dispatch_unlocked [inlined] g_main_dispatch(context=0x00003fffecc00010) at gmain.c:3344:27
    frame #33: 0x00003fffedd2b8b4 libglib-2.0.so.0`g_main_context_dispatch_unlocked(context=0x00003fffecc00010) at gmain.c:4152:7
    frame #34: 0x00003fffedd2c260 libglib-2.0.so.0`g_main_context_iterate_unlocked(context=0x00003fffecc00010, block=<unavailable>, dispatch=1, self=<unavailable>) at gmain.c:4217:5
    frame #35: 0x00003fffedd2cb6c libglib-2.0.so.0`g_main_loop_run(loop=0x00003fffebd00ef0) at gmain.c:4419:5
    frame #36: 0x00003fffeec739a8 libgtk-3.so.0`gtk_main at gtkmain.c:1329:7
    frame #37: 0x000000010002be9c xfce4-terminal`main(argc=<unavailable>, argv=<unavailable>) at main.c:349:3
    frame #38: 0x00003fffefa66008 libc.so`libc_start_main_stage2(main=(xfce4-terminal`main at main.c:174), argc=1, argv=0x00003ffffffff468) at __libc_start_main.c:95:7
    frame #39: 0x00003fffefa65f68 libc.so`__libc_start_main(main=<unavailable>, argc=1, argv=0x00003ffffffff468, init_dummy=<unavailable>, fini_dummy=<unavailable>, ldso_dummy=<unavailable>) at __libc_start_main.c:86:9
    frame #40: 0x000000010002b2f4 xfce4-terminal`_start_c(p=<unavailable>) at crt1.c:18:2
    frame #41: 0x000000010002b2a8 xfce4-terminal`_start + 40
(lldb) f 1
frame #1: 0x00003fffed92ed38 libgnutls.so.30`gnutls_cipher_set_iv [inlined] _gnutls_cipher_setiv(handle=0x00003fffecc0b0d0, iv=0x0000000000000000, ivlen=70368421260848) at cipher_int.h:77:9
(lldb) fr v
(const cipher_hd_st *) handle = 0x00003fffecc0b0d0
(const void *) iv = 0x0000000000000000
(size_t) ivlen = 70368421260848
(lldb)
AlbertoGP commented 1 month ago

Aside, what’s the executable for the Gnome Console application? I’d like to test launching it the other way around for completeness, Gnome Console from xfce4-terminal, but I can not figure out what the executable file for it is.

Update: Yes, launching kgx from xfce4-terminal through lldb and running ls -R /usr in kgx crashes in exactly the same way.

nekopsykose commented 1 month ago

Aside, what’s the executable for the Gnome Console application?

kgx. apk info -L gnome-console

nekopsykose commented 1 month ago

issue is probably the (size_t) ivlen = 70368421260848

nekopsykose commented 1 month ago

i'm gonna guess the casts inside gnutls between these types/functions causes some weird UB and so the numbers get fucked up, cause you can see the bt looks fine: 


    frame #1: 0x00003fffed92ed38 libgnutls.so.30`gnutls_cipher_set_iv [inlined] _gnutls_cipher_setiv(handle=0x00003fffecc0b0d0, iv=0x0000000000000000, ivlen=70368421260848) at cipher_int.h:77:9
    frame #2: 0x00003fffed92ed10 libgnutls.so.30`gnutls_cipher_set_iv(handle=0x00003fffecc0b0d0, iv=0x00003fffecc0aed8, ivlen=12) at crypto-api.c:188:6

note the 12 magically turns into garbage after a single inlined call

nekopsykose commented 1 month ago

hm, the iv=0x00003fffecc0aed8 is actually 70368421261016. which is curiously almost the swapped values..

nekopsykose commented 1 month ago

if you have a cports checkout and know how to rebuild the gnutls package, could you test building it with !lto, and maybe also tool_flags = {"CFLAGS": ["-fno-strict-aliasing"]}?

AlbertoGP commented 1 month ago

I won’t be able to do that today.

q66 commented 1 month ago

there is a good chance just !lto will fix it, i stared at the code a bit and this looks like a potential miscompilation, but with C you never know because there could be UB lurking with far-reaching consequences

mrkcee commented 1 month ago

I can also replicate this in aarch64. I'm running Chimera Linux on an aarch64 VM (UTM in MacOS). I open console (while in GNOME) then execute ls -l /usr/lib. It immediately crashes.

I have done and tested the following:

nekopsykose commented 1 month ago

seems to fix it for me on aarch64 but double check

mrkcee commented 1 month ago

Confirmed that the latest changes fixed the issue in aarch64.

AlbertoGP commented 4 weeks ago

Fixed also in ppc64le.