Closed acc15 closed 7 months ago
Well, if you rebuild tdesktop from source, can't you just build it (and perhaps Qt) with debug symbols, run inside gdb (gdb ./Telegram
, run
), interrupt (Ctrl+C) when it hangs and get a backtrace with bt
?
After i build qtbase and launch gdb with debug symbols:
LD_LIBRARY_PATH=/home/acc15/dev/thirdparty/qt/qtbase/build/lib gdb --args telegram-desktop -noupdate
It launches in 0.3 seconds...
Not telegram problem. Maybe distro packaging, as solution - rebuild QtBase.
After adding some additional logging, i've found that problem somewhere in Sandbox constructor (or as i guess - even deeper - QApplication):
Sandbox::start calls QCoreApplication::exec at its end whose intended behavior is to return only at application quit as it provides event loop. After event loop starts, the QLocalSocket signals should fire. QLocalSocket::errorOccurred calls Sandbox::socketError which in turn calls Sandbox::singleInstanceChecked which in turn calls Sandbox::launchApplication which creates the Core::Application instance.
Almost entire time the application should be blocked in poll() which interrupts when some event happens or timer exceeds, the code goes to processing events and then blocks in poll() again. That's the event loop.
LD_LIBRARY_PATH=/home/acc15/dev/thirdparty/qt/qtbase/build/lib
Wouldn't rebuilding the distro Qt package with debug symbols be a better test? I even heard Arch provides debug info for Qt via debuginfod...
Yeah, thanks for hints and explanation!
I finally find place where all that time was wasted... it spins in proprietary NVIDIA lib:
(gdb) where
#0 0x00007ffff0985489 in ??? () at /usr/lib/libc.so.6
#1 0x0000555555c50d9f in realloc ()
#2 0x00007fd2db1e8ff4 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#3 0x00007fd2db1ffa61 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#4 0x00007fd2db74fa98 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#5 0x00007fd2db1f4d06 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#6 0x00007fd2dc91ab6a in ??? () at /usr/lib/libGLX_nvidia.so.0
#7 0x00007fd2dc974692 in ??? () at /usr/lib/libGLX_nvidia.so.0
#8 0x00007fffffffc6f0 in ??? ()
#9 0x00007fdedf23f410 in ??? ()
#10 0x0000000000000002 in ??? ()
#11 0x00007fd2dc91a013 in ??? () at /usr/lib/libGLX_nvidia.so.0
#12 0x00007fdedf23f410 in ??? ()
#13 0x00007ffff7fcd175 in ??? () at /lib64/ld-linux-x86-64.so.2
#14 0x00007ffff7fcd2ac in ??? () at /lib64/ld-linux-x86-64.so.2
#15 0x00007ffff7fc956e in _dl_catch_exception () at /lib64/ld-linux-x86-64.so.2
#16 0x00007ffff7fd3d26 in ??? () at /lib64/ld-linux-x86-64.so.2
#17 0x00007ffff7fc94e1 in _dl_catch_exception () at /lib64/ld-linux-x86-64.so.2
#18 0x00007ffff7fd410c in ??? () at /lib64/ld-linux-x86-64.so.2
#19 0x00007ffff08a53fc in ??? () at /usr/lib/libc.so.6
#20 0x00007ffff7fc94e1 in _dl_catch_exception () at /lib64/ld-linux-x86-64.so.2
#21 0x00007ffff7fc9603 in ??? () at /lib64/ld-linux-x86-64.so.2
#22 0x00007ffff08a4f07 in ??? () at /usr/lib/libc.so.6
#23 0x00007ffff08a54b1 in dlopen () at /usr/lib/libc.so.6
#24 0x00007fffeded7bc7 in ??? () at /usr/lib/libGLX.so.0
#25 0x00007fffededa415 in ??? () at /usr/lib/libGLX.so.0
#26 0x00007fffededb24c in glXChooseFBConfig () at /usr/lib/libGLX.so.0
#27 0x00007ffff1f56a05 in qglx_findConfig(_XDisplay*, int, QSurfaceFormat, bool, int, int) () at /usr/lib/libQt6Gui.so.6
#28 0x00007fd2decdfcc7 in ??? () at /usr/lib/qt6/plugins/xcbglintegrations/libqxcb-glx-integration.so
#29 0x00007fd2dece0f75 in ??? () at /usr/lib/qt6/plugins/xcbglintegrations/libqxcb-glx-integration.so
#30 0x00007ffff1e80520 in QOpenGLContext::create() () at /usr/lib/libQt6Gui.so.6
#31 0x00007fd2dec9782e in ??? () at /usr/lib/qt6/plugins/platformthemes/KDEPlasmaPlatformTheme6.so
#32 0x00007ffff0f3da5a in QCoreApplicationPrivate::init() () at /usr/lib/libQt6Core.so.6
#33 0x00007ffff1b74ac1 in QGuiApplicationPrivate::init() () at /usr/lib/libQt6Gui.so.6
#34 0x00007ffff2cf53ee in QApplicationPrivate::init() () at /usr/lib/libQt6Widgets.so.6
#35 0x0000555556be7b4b in ??? ()
#36 0x0000555556be9b6e in ??? ()
#37 0x0000555555c2e9ff in ??? ()
#38 0x00007ffff0843cd0 in ??? () at /usr/lib/libc.so.6
#39 0x00007ffff0843d8a in __libc_start_main () at /usr/lib/libc.so.6
#40 0x0000555555c470d5 in ??? ()
also checked strace - billions of mmap, munmap and nothing more...
it spins in proprietary NVIDIA lib:
Well, not exactly. It goes through the NVIDIA library and the Google's scudo allocator bundled within tdesktop to some glibc function which is sadly unknown due to the lack of glibc debug symbols.
Hmmm, i build telegram without scudo allocator
-D DESKTOP_APP_DISABLE_SCUDO=ON \
And problem gone, fast startup using packaged qt and all other libs...
And finally finally, it looks like the problem is scudo
+ nvidia
.
Also attaching strace - what's actually happening here i'm still dont understand due to nvidia closed sources, it do realloc, scudo do mass mmap
, munmap
without any progress strace_telegram.txt
well, if you had debug symbols for tdesktop and glibc, it would be much simpler as you would have the file and line number for the frames. Maybe it hangs in some kind of mutex, conditional variable or etc.
#0 __memcpy_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:660
#1 0x0000555555c50d9f in memcpy (__len=5938352, __src=0x7fd2d7601350, __dest=0x7fd2d6801320) at /usr/include/bits/string_fortified.h:29
#2 scudo::Allocator<scudo::DefaultConfig, &malloc_postinit>::reallocate (Alignment=16, NewSize=5942496, OldPtr=0x7fd2d7601350, this=0x55555b22cd80 <Allocator>) at /usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/ThirdParty/scudo/combined.h:699
#3 realloc (ptr=0x7fd2d7601350, size=5942496) at /usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/ThirdParty/scudo/wrappers_c.inc:150
#4 0x00007fd2db1e8ff4 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#5 0x00007fd2db1ffa61 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#6 0x00007fd2db74fa98 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#7 0x00007fd2db1f4d06 in ??? () at /usr/lib/libnvidia-glcore.so.550.54.14
#8 0x00007fd2dc925b6a in ??? () at /usr/lib/libGLX_nvidia.so.0
#9 0x00007fd2dc97f692 in ??? () at /usr/lib/libGLX_nvidia.so.0
#10 0x00007fffffffc6f0 in ??? ()
#11 0x00007fd3df23b710 in ??? ()
#12 0x0000000000000002 in ??? ()
#13 0x00007fd2dc925013 in ??? () at /usr/lib/libGLX_nvidia.so.0
#14 0x00007fd3df23b710 in ??? ()
#15 0x00007ffff7fcd175 in call_init (env=0x7fd2df210c10, argv=0x7fffffffda48, argc=-15456, l=0x7fd2dc9d6f98) at dl-init.c:60
#16 call_init (l=0x7fd2dc9d6f98, argc=-15456, argv=0x7fffffffda48, env=0x7fd2df210c10) at dl-init.c:26
#17 0x00007ffff7fcd2ac in _dl_init (main_map=0x7fd3df23b710, argc=2, argv=0x7fffffffda48, env=0x7fd2df210c10) at dl-init.c:121
#18 0x00007ffff7fc956e in __GI__dl_catch_exception (exception=exception@entry=0x0, operate=operate@entry=0x7ffff7fd3d80 <call_dl_init>, args=args@entry=0x7fffffffc550) at dl-catch.c:211
#19 0x00007ffff7fd3d26 in dl_open_worker (a=a@entry=0x7fffffffc6f0) at dl-open.c:829
#20 0x00007ffff7fc94e1 in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffc6d0, operate=operate@entry=0x7ffff7fd3c90 <dl_open_worker>, args=args@entry=0x7fffffffc6f0) at dl-catch.c:237
#21 0x00007ffff7fd410c in _dl_open (file=0x7ffbdf212f10 "libGLX_nvidia.so.0", mode=<optimized out>, caller_dlopen=0x7fffe5527bc7 <__glXLookupVendorByName+1271>, nsid=<optimized out>, argc=2, argv=0x7fffffffda48, env=0x7fd2df210c10) at dl-open.c:905
#22 0x00007ffff08a53fc in dlopen_doit (a=a@entry=0x7fffffffc960) at dlopen.c:56
#23 0x00007ffff7fc94e1 in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffc8c0, operate=0x7ffff08a53a0 <dlopen_doit>, args=0x7fffffffc960) at dl-catch.c:237
#24 0x00007ffff7fc9603 in _dl_catch_error (objname=0x7fffffffc918, errstring=0x7fffffffc920, mallocedp=0x7fffffffc917, operate=<optimized out>, args=<optimized out>) at dl-catch.c:256
#25 0x00007ffff08a4f07 in _dlerror_run (operate=operate@entry=0x7ffff08a53a0 <dlopen_doit>, args=args@entry=0x7fffffffc960) at dlerror.c:138
#26 0x00007ffff08a54b1 in dlopen_implementation (dl_caller=<optimized out>, mode=<optimized out>, file=<optimized out>) at dlopen.c:71
#27 ___dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:81
#28 0x00007fffe5527bc7 in __glXLookupVendorByName (vendorName=<optimized out>) at ../libglvnd-v1.7.0/src/GLX/libglxmapping.c:428
#29 0x00007fffe552a415 in __glXLookupVendorByScreen (dpy=dpy@entry=0x7fe3df210c10, screen=screen@entry=0) at ../libglvnd-v1.7.0/src/GLX/libglxmapping.c:572
#30 0x00007fffe552b24c in __glXGetDynDispatch (screen=0, dpy=0x7fe3df210c10) at ../libglvnd-v1.7.0/src/GLX/libglxmapping.c:606
#31 glXChooseFBConfig (dpy=0x7fe3df210c10, screen=0, attrib_list=0x7feadf2809e0, nelements=0x7fffffffcb48) at ../libglvnd-v1.7.0/src/GLX/libglx.c:1484
#32 0x00007ffff1f56a05 in qglx_findConfig (display=0x7fe3df210c10, screen=0, format=..., highestPixelFormat=highestPixelFormat@entry=false, drawableBit=drawableBit@entry=1, flags=flags@entry=0)
at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/corelib/tools/qarraydatapointer.h:114
#33 0x00007fd2deceacc7 in QGLXContext::QGLXContext (this=0x7ff8df22d510, display=<optimized out>, screen=0x7fe7df202210, format=<optimized out>, share=0x0, this=<optimized out>, display=<optimized out>, screen=<optimized out>, format=<optimized out>, share=<optimized out>)
at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/plugins/platforms/xcb/gl_integrations/xcb_glx/qglxintegration.cpp:201
#34 0x00007fd2decebf75 in QXcbGlxIntegration::createPlatformOpenGLContext (this=0x7fdddf26fc90, context=<optimized out>) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/plugins/platforms/xcb/gl_integrations/xcb_glx/qxcbglxintegration.cpp:158
#35 0x00007ffff1e80520 in QOpenGLContext::create (this=this@entry=0x7fffffffce20) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/gui/kernel/qopenglcontext.cpp:355
#36 0x00007fd2deca282e in checkBackend (checkContext=...) at /usr/src/debug/plasma-integration/plasma-integration-6.0.2/qt6/src/platformtheme/qtquickrenderersettings.cpp:43
#37 initializeRendererSessions () at /usr/src/debug/plasma-integration/plasma-integration-6.0.2/qt6/src/platformtheme/qtquickrenderersettings.cpp:66
#38 0x00007ffff0f3da5a in qt_call_pre_routines () at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/corelib/kernel/qcoreapplication.cpp:306
#39 QCoreApplicationPrivate::init (this=this@entry=0x7fe7df202410) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/corelib/kernel/qcoreapplication.cpp:914
#40 0x00007ffff1b74ac1 in QGuiApplicationPrivate::init (this=this@entry=0x7fe7df202410) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/gui/kernel/qguiapplication.cpp:1579
#41 0x00007ffff2cf53ee in QApplicationPrivate::init (this=0x7fe7df202410) at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/widgets/kernel/qapplication.cpp:473
#42 0x00007ffff2cf5542 in QApplication::QApplication (this=this@entry=0x7fffffffd400, argc=@0x7fffffffd3c0: 1, argv=<optimized out>, this=<optimized out>, argc=<optimized out>, argv=<optimized out>)
at /usr/src/debug/qt6-base/qtbase-everywhere-src-6.6.2/src/widgets/kernel/qapplication.cpp:461
#43 0x0000555556be7b4b in Core::Sandbox::Sandbox (argv=<optimized out>, argc=@0x7fffffffd3c0: 1, this=0x7fffffffd400) at /usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/SourceFiles/core/sandbox.cpp:83
#44 Core::Launcher::executeApplication (this=this@entry=0x7fefdf223e90) at /usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/SourceFiles/core/launcher.cpp:579
#45 0x0000555556be9b6e in Core::Launcher::exec (this=0x7fefdf223e90) at /usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/SourceFiles/core/launcher.cpp:388
#46 0x0000555555c2e9ff in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/SourceFiles/main.cpp:12
calls to realloc (/usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/ThirdParty/scudo/wrappers_c.inc:150) from nvidia with bigger and bigger sizes...
ptr = 0x7fd2d6601bd0
size = 6108256
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $8 = (void *) 0x7fd2d6001ba0
(gdb) c
Continuing.
ptr = 0x7fd2d6001ba0
size = 6112400
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $9 = (void *) 0x7fd2d6c01b70
(gdb) c
Continuing.
ptr = 0x7fd2d6c01b70
size = 6116544
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $10 = (void *) 0x7fd2d6601b40
(gdb) c
Continuing.
ptr = 0x7fd2d6601b40
size = 6120688
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $11 = (void *) 0x7fd2d6001b10
(gdb) c
Continuing.
ptr = 0x7fd2d6001b10
size = 6124832
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $12 = (void *) 0x7fd2d6c01ae0
(gdb) c
Continuing.
ptr = 0x7fd2d6c01ae0
size = 6128976
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $13 = (void *) 0x7fd2d6601ab0
(gdb) c
Continuing.
ptr = 0x7fd2d6601ab0
size = 6133120
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $14 = (void *) 0x7fd2d6001a80
(gdb) c
Continuing.
ptr = 0x7fd2d6001a80
size = 6137264
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $15 = (void *) 0x7fd2d6c01a50
(gdb) c
Continuing.
ptr = 0x7fd2d6c01a50
size = 6141408
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $16 = (void *) 0x7fd2d6601a20
(gdb) c
Continuing.
ptr = 0x7fd2d6601a20
size = 6145552
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $17 = (void *) 0x7fd2d60019f0
(gdb) c
Continuing.
ptr = 0x7fd2d60019f0
size = 6149696
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $18 = (void *) 0x7fd2d6c019c0
(gdb) c
Continuing.
ptr = 0x7fd2d6c019c0
size = 6153840
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $19 = (void *) 0x7fd2d6601990
(gdb) c
Continuing.
ptr = 0x7fd2d6601990
size = 6157984
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $20 = (void *) 0x7fd2d6001960
(gdb) c
Continuing.
ptr = 0x7fd2d6001960
size = 6162128
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $21 = (void *) 0x7fd2d6c01930
(gdb) c
Continuing.
ptr = 0x7fd2d6c01930
size = 6166272
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $22 = (void *) 0x7fd2d6601900
(gdb) c
Continuing.
ptr = 0x7fd2d6601900
size = 6170416
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $23 = (void *) 0x7fd2d60018d0
(gdb) c
Continuing.
ptr = 0x7fd2d60018d0
size = 6174560
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $24 = (void *) 0x7fd2d6c018a0
(gdb) c
Continuing.
ptr = 0x7fd2d6c018a0
size = 6178704
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $25 = (void *) 0x7fd2d6601870
(gdb) c
Continuing.
ptr = 0x7fd2d6601870
size = 6182848
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $26 = (void *) 0x7fd2d6001840
(gdb) c
Continuing.
ptr = 0x7fd2d6001840
size = 6186992
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $27 = (void *) 0x7fd2d6c01810
(gdb) c
Continuing.
ptr = 0x7fd2d6c01810
size = 6191136
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $28 = (void *) 0x7fd2d66017e0
(gdb) c
Continuing.
ptr = 0x7fd2d66017e0
size = 6195280
0x00007fd2db1e8ff4 in ?? () from /usr/lib/libnvidia-glcore.so.550.54.14
Value returned is $29 = (void *) 0x7fd2d60017b0
Oh well, maybe scudo is just not performant enough for NVIDIA then, it's a hardened allocator after all. NVIDIA seems to do a weird thing, though...
I don't really see a way out of this. glibc's allocator doesn't really free memory with tdesktop allocation pattern due to unknown reason, so it's either long slow startup with NVIDIA or leaking for everyone.
the dev branch has a newer scudo btw, feel free to try it, maybe it's better
The same behavior when using glibc. But its just faster. I'll try to report this to NVIDIA...
2 scudo::Allocator<scudo::DefaultConfig, &malloc_postinit>::reallocate (Alignment=16, NewSize=5942496, OldPtr=0x7fd2d7601350, this=0x55555b22cd80
) at /usr/src/debug/telegram-desktop/tdesktop-4.15.2-full/Telegram/ThirdParty/scudo/combined.h:699
It has a funny comment: https://github.com/telegramdesktop/tdesktop/blob/82fda96ed88065a761e27cb80173250c81aba8a9/Telegram/ThirdParty/scudo/combined.h#L693-L696 Perhaps this should also be reported to https://github.com/llvm/llvm-project/issues that there's real gain.
It looks like the new NVIDIA 550.67 fixes this
are you sure? tdesktop reverted to jemalloc after all
It looks like the new NVIDIA 550.67 fixes this
it doesn't. still hangs on startup with scudo allocator
Steps to reproduce
If i use Wayland session - everything is OK - it starts almost instantly. Problem is X11 related I've tested both GNOME and KDE on X11 - result is same - slow startup on X11, fast startup on Wayland.
Also i've tried binary from official site. Tried to rebuild release from github sources.
After adding some additional logging, i've found that problem somewhere in Sandbox constructor (or as i guess - even deeper - QApplication):
Expected behaviour
Startup time is about 1-2 seconds
Actual behaviour
Startup time is about 25 seconds
Operating system
ArchLinux, KDE, X11
Version of Telegram Desktop
4.15.2
Installation source
Static binary from official website
Crash ID
No response
Logs