Native run on PyQt-WebEngine hangs with jemalloc

toofar commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Hello, me again! Now that I can do native runs with PyQt WebEngine I went back to some scenarios I've tested previously with a non-native run (for context I'm looking into https://github.com/qutebrowser/qutebrowser/issues/1476). I'm looking into how different memory allocaters might make the memory load of a particular application a bit lighter.

When running an application based on PyQt WebEngine with the jemalloc library in LD_PRELOAD the run hangs. With other mallocs (glibc, mimalloc, tcmalloc) the run completes fine.

Here's the stack traces, it seems to be hung up on a semaphore? Which sounds like it could be some fun timing thing. Also it looks like it is coming from DBus related code so probably it doesn't necessarily need all of QtWebEngine to reproduce.

(gdb) thread apply all bt


Thread 3 (Thread 0x7f09d24236c0 (LWP 18456) "python3"):
#0  futex_wait (private=0, expected=2, futex_word=0x7f09e86d5700) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7f09e86d5700, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f09e8382262 in lll_mutex_lock_optimized (mutex=0x7f09e86d5700) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7f09e86d5700) at ./nptl/pthread_mutex_lock.c:93
#4  0x00007f09e86923a0 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#5  0x00007f09e8621944 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#6  0x00007f09e8621b8f in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#7  0x00007f09e862259d in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#8  0x00007f09e86addf9 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#9  0x00007f09e6eedaea in memray::tracking_api::Tracker::prepareNativeTrace (trace=std::optional [no contained value]) at src/memray/_memray/tracking_api.h:237
#10 0x00007f09e6eeed11 in memray::tracking_api::Tracker::trackAllocation (func=memray::hooks::Allocator::MMAP, size=2097152, ptr=0x7f09de600000) at src/memray/_memray/tracking_api.h:218
#11 memray::intercept::mmap (addr=, length=2097152, prot=, flags=, fd=, offset=) at s--Type  for more, q to quit, c to continue without paging--c
rc/memray/_memray/hooks.cpp:224
#12 0x00007f09e8694b5a in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#13 0x00007f09e8694bc2 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#14 0x00007f09e8689994 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#15 0x00007f09e863bc89 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#16 0x00007f09e863cc5f in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#17 0x00007f09e863769c in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#18 0x00007f09e8621878 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#19 0x00007f09e86abc5a in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#20 0x00007f09e86abef8 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#21 0x00007f09e86ad926 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#22 0x00007f09e8622162 in ?? () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
#23 0x00007f09e6eee4c5 in memray::hooks::SymbolHook::operator()(unsigned long) const (this=0x7f09e6f6c6c0 ) at src/memray/_memray/hooks.h:100
#24 memray::intercept::malloc (size=168) at src/memray/_memray/hooks.cpp:169
#25 0x00007f09e8935ece in malloc (size=) at ../include/rtld-malloc.h:56
#26 allocate_dtv_entry (size=, alignment=16) at ../elf/dl-tls.c:684
#27 allocate_and_init (map=0x7f09e6161b00) at ../elf/dl-tls.c:709
#28 tls_get_addr_tail (ti=0x7f09e3a0a9c0, dtv=0x7f09e621f410, the_map=0x7f09e6161b00) at ../elf/dl-tls.c:907
#29 0x00007f09e8939468 in __tls_get_addr () at ../sysdeps/x86_64/tls_get_addr.S:55
#30 0x00007f09e36cfaa9 in QThreadPrivate::start(void*) () from /usr/local/Qt-6.5.0/lib/libQt6Core.so.6
#31 0x00007f09e837efd4 in start_thread (arg=) at ./nptl/pthread_create.c:442
#32 0x00007f09e83ff66c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f09e47ff6c0 (LWP 18455) "python3"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f09e47fe410, op=393, expected=0, futex_word=0x7f09e61df468) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f09e61df468, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f09e47fe410, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f09e837bd9b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f09e61df468, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f09e47fe410, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f09e837e6dc in __pthread_cond_wait_common (abstime=0x7f09e47fe410, clockid=0, mutex=0x7f09e61df418, cond=0x7f09e61df440) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_timedwait64 (cond=0x7f09e61df440, mutex=0x7f09e61df418, abstime=0x7f09e47fe410) at ./nptl/pthread_cond_wait.c:643
#5  0x00007f09e6f0b3cf in __gthread_cond_timedwait (__abs_timeout=0x7f09e47fe410, __mutex=, __cond=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/x86_64-redhat-linux/bits/gthr-default.h:872
#6  std::condition_variable::__wait_until_impl > > (__lock=..., __lock=..., __atime=..., this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/condition_variable:232
#7  std::condition_variable::wait_until > > (__atime=..., __lock=..., this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/condition_variable:141
#8  std::condition_variable::wait_until >, memray::tracking_api::Tracker::BackgroundThread::start():::: > (__p=..., __atime=..., __lock=..., this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/condition_variable:158
#9  std::condition_variable::wait_for, memray::tracking_api::Tracker::BackgroundThread::start():::: > (__rtime=..., __rtime=..., __p=..., __lock=..., this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/condition_variable:185
#10 operator() (__closure=) at src/memray/_memray/tracking_api.cpp:690
#11 std::__invoke_impl > (__f=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/invoke.h:60
#12 std::__invoke > (__fn=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/invoke.h:95
#13 std::thread::_Invoker > >::_M_invoke<0> (this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/thread:264
#14 std::thread::_Invoker > >::operator() (this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/thread:271
#15 std::thread::_State_impl > > >::_M_run(void) (this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/thread:215
#16 0x00007f09e80d44a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007f09e837efd4 in start_thread (arg=) at ./nptl/pthread_create.c:442
#18 0x00007f09e83ff66c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f09e88fc5c0 (LWP 18454) "python3"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f09e36d16ce in QSemaphore::acquire(int) () from /usr/local/Qt-6.5.0/lib/libQt6Core.so.6
#2  0x00007f09e3596d36 in void doActivate(QObject*, int, void**) () from /usr/local/Qt-6.5.0/lib/libQt6Core.so.6
#3  0x00007f09e57730ed in QDBusConnectionManager::connectToBus(QDBusConnection::BusType, QString const&, bool) () from /usr/local/Qt-6.5.0/lib/libQt6DBus.so.6
#4  0x00007f09e5773227 in QDBusConnectionManager::busConnection(QDBusConnection::BusType) () from /usr/local/Qt-6.5.0/lib/libQt6DBus.so.6
#5  0x00007f09e5774748 in QDBusConnection::sessionBus() () from /usr/local/Qt-6.5.0/lib/libQt6DBus.so.6
#6  0x00007f09dfc9efc1 in QGenericUnixServices::QGenericUnixServices() () from /usr/local/Qt-6.5.0/lib/libQt6Gui.so.6
#7  0x00007f09def29390 in QXcbIntegration::QXcbIntegration(QList const&, int&, char**) () from /usr/local/Qt-6.5.0/plugins/platforms/../../lib/libQt6XcbQpa.so.6
#8  0x00007f09e2ff13dc in QXcbIntegrationPlugin::create(QString const&, QList const&, int&, char**) () from /usr/local/Qt-6.5.0/plugins/platforms/libqxcb.so
#9  0x00007f09df7b2584 in init_platform(QString const&, QString const&, QString const&, int&, char**) () from /usr/local/Qt-6.5.0/lib/libQt6Gui.so.6
#10 0x00007f09df7b5ca1 in QGuiApplicationPrivate::createPlatformIntegration() () from /usr/local/Qt-6.5.0/lib/libQt6Gui.so.6
#11 0x00007f09df7b6810 in QGuiApplicationPrivate::createEventDispatcher() () from /usr/local/Qt-6.5.0/lib/libQt6Gui.so.6
#12 0x00007f09e3540a55 in QCoreApplicationPrivate::init() () from /usr/local/Qt-6.5.0/lib/libQt6Core.so.6
#13 0x00007f09df7b99f9 in QGuiApplicationPrivate::init() () from /usr/local/Qt-6.5.0/lib/libQt6Gui.so.6
#14 0x00007f09e4b830f9 in QApplicationPrivate::init() () from /usr/local/Qt-6.5.0/lib/libQt6Widgets.so.6
#15 0x00007f09e552b939 in sipQApplication::sipQApplication(int&, char**, int) () from /mnt/ssd/user/qute-basedirs/venv-6.5/lib/python3.11/site-packages/PyQt6/QtWidgets.abi3.so
#16 0x00007f09e552ba07 in init_type_QApplication () from /mnt/ssd/user/qute-basedirs/venv-6.5/lib/python3.11/site-packages/PyQt6/QtWidgets.abi3.so
#17 0x00007f09e6a3e7d2 in sipSimpleWrapper_init (self=0x7f09e5fa2320, args=0x7f09e64b5a20, kwds=0x0) at sip_core.c:8788
#18 0x0000000000518b75 in _PyObject_MakeTpCall ()
#19 0x00000000005372e0 in _PyEval_EvalFrameDefault ()
#20 0x00000000005242fb in PyEval_EvalCode ()
#21 0x000000000058fdde in ?? ()
#22 0x000000000053bb47 in ?? ()
#23 0x000000000053ba7c in PyObject_Vectorcall ()
#24 0x000000000053741b in _PyEval_EvalFrameDefault ()
#25 0x00000000005242fb in PyEval_EvalCode ()
#26 0x000000000058fdde in ?? ()
#27 0x000000000053bb47 in ?? ()
#28 0x000000000053ba7c in PyObject_Vectorcall ()
#29 0x000000000052c5f0 in _PyEval_EvalFrameDefault ()
#30 0x000000000055da51 in _PyFunction_Vectorcall ()
#31 0x00000000006523bd in ?? ()
#32 0x0000000000651e47 in Py_RunMain ()
#33 0x0000000000629b47 in Py_BytesMain ()
#34 0x00007f09e831d18a in __libc_start_call_main (main=main@entry=0x629ab0, argc=argc@entry=6, argv=argv@entry=0x7ffd329232a8) at ../sysdeps/nptl/libc_start_call_main.h:58
#35 0x00007f09e831d245 in __libc_start_main_impl (main=0x629ab0, argc=6, argv=0x7ffd329232a8, init=, fini=, rtld_fini=, stack_end=0x7ffd32923298) at ../csu/libc-start.c:381
#36 0x00000000006299e1 in _start ()

And here is a script to reproduce the freeze in a container (remove the --native or the whole -m memray run --native bits to see it run through fully for all mallocs):

docker run -i debian:unstable bash -s <<"EOF"
cat >testbrowser_webengine.py <<"EOP"
#!/usr/bin/env python3
import sys
from PyQt6.QtCore import QUrl, QTimer
from PyQt6.QtWidgets import QApplication
from PyQt6.QtWebEngineWidgets import QWebEngineView

app = QApplication(sys.argv)
wv = QWebEngineView()

def kill():
    app.quit()
QTimer.singleShot(10, kill)

wv.load(QUrl.fromUserInput("https://example.com/"))
wv.show()

app.exec()
EOP
apt update
apt install -y wget xvfb python3-venv python3-pyqt6.qtwebengine libjemalloc2 libmimalloc2.0 libtcmalloc-minimal4
export DEBUGINFOD_URLS=https://debuginfod.debian.net
export QTWEBENGINE_CHROMIUM_FLAGS=--no-sandbox  # running as root in container
python3 -m venv --system-site-packages memray-test
. memray-test/bin/activate
# pip3 install memray  # need fix from #334
# I uploaded the wheel here because JS seems to be required to DL from GH actions
wget -O memray-1.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl https://files.catbox.moe/xglmxq.whl
pip3 install memray-1.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
# run with a few mallocs, it works fine with all of them except jemalloc
for lib in "" /usr/lib/x86_64-linux-gnu/libmimalloc.so.2 /usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4 /usr/lib/x86_64-linux-gnu/libjemalloc.so.2 ;do
  export LD_PRELOAD=$lib
  printf "\n=====\nRunning with LD_PRELOAD=$LD_PRELOAD\n=====\n\n"
  xvfb-run python3 -m memray run --native testbrowser_webengine.py  # hangs here w/ jemalloc
  echo "Finished run, generating flamegraph"
  python3 -m memray flamegraph `ls -tr *.bin | tail -n1`
done
EOF

Expected Behavior

No response

Steps To Reproduce

see above

Memray Version

latest build from https://github.com/bloomberg/memray/actions/runs/4427648999

Python Version

3.11

Operative System

Linux

Anything else?

No response

godlygeek commented 1 year ago

Hm. At a glance, without firing up a debugger, I'm thinking that jemalloc's malloc isn't reentrant, and we're calling back into it at a time that it wasn't prepared for. I'll try to dig into it in a day or so...