thread sanitizer leaking

[grandinj]

I'm running TSAN over the LibreOffice codebase, and with certain unit tests, clang will basically use up all available memory on my box (16G) before the OOM killer kicks in.

I'm using TSAN_OPTIONS=suppressions=/media/disk2/libo-san/tsan-suppress.txt:memory_limit_mb=4096:flush_memory_ms=250

and I'm building with clang -fsanitize=thread

and I'm running clang from trunk about 4 weeks ago.

What can I do to reduce the memory usage or track down the bug?

[dvyukov]

Hi Noel,

Is it clang consumes all memory, or your program consumes all memory?

[grandinj]

My apologies, it is the program consuming all memory.

Under normal conditions (i.e. without thread sanitizer), this unit test has a maximum memory consumption of around 100M

[grandinj]

Ah, I see you are working on a new backend (V3)

I'll just wait for that to land, and retest then rather than waste both our time debugging the older backend (or possibly some pathological behaviour in the Libreoffice code)

[dvyukov]

The memory usage increase from ~100M to ~16GB is somewhat expected. The simplest thing to try is to run with tsan memory profiler as:

TSAN_OPTIONS=profile_memory=/tmp/tsan.memprof ./myprogram

It may or may not shed some light onto the nature of the problem.

Ah, I see you are working on a new backend (V3)

Yes. It should reduce memory consumption of programs and any issues will now be fixed only in the new version (if still persist). It's actually ready for submission, it needs a code review, but I don't have any remaining TODOs. If you feel adventurous you may checkout https://github.com/dvyukov/llvm-project/tree/tsanv3 and try it. A testing result (crash/don't crash) on a large real program will be very useful as well.

[grandinj]

The last few lines of the memprof file are:

25s: RSS 860 MB: shadow:334 meta:36 file:173 mmap:41 trace:4 heap:74 other:0 intalloc:3 memblocks:9 syncobj:31 stacks=151[293092] nthr=1/12 25s: RSS 861 MB: shadow:334 meta:36 file:173 mmap:41 trace:4 heap:74 other:0 intalloc:3 memblocks:9 syncobj:31 stacks=151[293092] nthr=1/12 26s: RSS 850 MB: shadow:330 meta:36 file:170 mmap:42 trace:4 heap:69 other:0 intalloc:3 memblocks:9 syncobj:31 stacks=152[293830] nthr=1/12

So it looks like it's the stacks that are growing without bound.

[dvyukov]

Just to make sure: this is only 850MB RSS and you mentioned 16GB. Do stacks grow up to gigs, while other numbers stay ~the same? 150MB is not a huge consumption on its own.

If it's stacks, then I afraid the new runtime won't help. It memorizes stacks the same way. But they are not supposed to grow w/o bound.

Re further debugging, it would be useful if stack depot could print every n-th stack memorized when asked with some flag. I guess every one of us implemented such patch locally several times already :) but we don't have it upstream... A typical root cause is that there is an unbounded number of legitimate or not-legitimate unique stack traces, e.g. signals, or some recursion, or tricks with context switches. But hard to say w/o seeing these stacks.

[grandinj]

So when I said 16GB at the beginning of this bug, I meant that the RSS grows to that.

Each of our executable unit tests contains a number of tests inside it. I can execute the unit test with only a subset of the child tests inside it.

For example, running

make CppunitTest_sw_ooxmlexport8 CPPUNIT_TEST_NAME="testFdo43093 testChartProp"

gets me

35s: RSS 1123 MB: shadow:341 meta:36 file:204 mmap:49 trace:3 heap:73 other:0 intalloc:2 memblocks:9 syncobj:32 stacks=370[580347] nthr=1/16
37s: RSS 1108 MB: shadow:338 meta:36 file:194 mmap:50 trace:3 heap:69 other:0 intalloc:2 memblocks:9 syncobj:32 stacks=371[581046] nthr=1/16

and running

make CppunitTest_sw_ooxmlexport8 CPPUNIT_TEST_NAME="testFdo43093 testChartProp testFdo69636 testVmlTextVerticalAdjust testFdo68607"

so that it 5 out of the 84 child tests, and that gets me

78s: RSS 1567 MB: shadow:413 meta:45 file:203 mmap:71 trace:2 heap:81 other:0 intalloc:2 memblocks:10 syncobj:40 stacks=696[1093208] nthr=1/19
80s: RSS 1549 MB: shadow:410 meta:45 file:193 mmap:71 trace:2 heap:76 other:0 intalloc:2 memblocks:10 syncobj:40 stacks=697[1093910] nthr=1/19

So it certainly looks like as I add more child tests the stacks memory grows.

Is it possible that the stacks memory is growing because the number of unique threads grows? Each child test will spawn a handful of threads (between 2 and 5) and then shut them down.

And I am guessing that the filtering for suppressions only happens at program termination?

[grandinj]

Actually, I don't know what the actual RSS would grow to, because my machine always dies before the unit test runs to completion :-)

[dvyukov]

I see. "stacks=697[1093910]" is indeed concerning.

No, it's not threads. It's unique stack traces. All sanitizers capture stack traces at some points like malloc, free, thread creation, mutex creation and memorize them with deduplication. Normally they don't grow w/o bound because number of unique stacks in the program is bounded.

And I am guessing that the filtering for suppressions only happens at program termination?

I don't understand how suppressions are related. Suppressions filter out race reports and they are applied during race reporting. But they have nothing to do with stack trace memorization.

[grandinj]

Is there a way I can dump all the stacks? Perhaps the de-duplication is running into issues?

[dvyukov]

There is no feature to dump stacks. I think we need something along the following lines:

--- a/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc
@@ -87,6 +87,9 @@ COMMON_FLAG(int, print_module_map, 0,
             "Print the process module map where supported (0 - don't print, "
             "1 - print only once before process exits, 2 - print after each "
             "report).")
+COMMON_FLAG(int, sample_unique_stacks, 0,
+            "Print every N-th stack added to stackdepot (0 - don't print). "
+            "Useful for debugging of runaway stackdepot memory consumption."
 COMMON_FLAG(bool, check_printf, true, "Check printf arguments.")
 #define COMMON_FLAG_HANDLE_SIGNAL_HELP(signal) \
     "Controls custom tool's " #signal " handler (0 - do not registers the " \
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_stackdepotbase.h b/compiler-rt/lib/sanitizer_common/sanitizer_stackdepotbase.h
index 96d1ddc87fd0..002bcfaf29d0 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_stackdepotbase.h
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_stackdepotbase.h
@@ -22,6 +22,8 @@

 namespace __sanitizer {

+void PrintMutexPC(uptr pc);
+
 template <class Node, int kReservedBits, int kTabSizeLog>
 class StackDepotBase {
   static constexpr u32 kIdSizeLog =
@@ -144,6 +146,10 @@ u32 StackDepotBase<Node, kReservedBits, kTabSizeLog>::Put(args_type args,
   new_node.link = s2;
   unlock(p, s);
   if (inserted) *inserted = true;
+  if (UNLIKELY(common_flags()->sample_unique_stacks) &&
+     (s % common_flags()->sample_unique_stacks) == 0) {
+     ... print the stack
+  }
   return s;
 }

[grandinj]

So I figured out the rest of the unique-stacks patch and then ran

export TSAN_OPTIONS=suppressions=/media/disk2/libo-tsan/tsan-suppress.txt:memory_limit_mb=4096:flush_memory_ms=250
TSAN_OPTIONS=$TSAN_OPTIONS:profile_memory=/media/disk2/libo-tsan/tsan.memprof:sample_unique_stacks=32 make CppunitTest_sw_ooxmlexport8 CPPUNIT_TEST_NAME="testFdo43093 testChartProp testFdo69636"

which resulted in this output:

https://www.icloud.com/iclouddrive/0ckw8_XVR49JSz9c50eaPWkkw#sw_ooxmlexport8.test

[dvyukov]

Thanks! Unfortunately I don't see anything wrong in the log. It's just lots of deep stacks, some subset I look at look perfectly legit. But there are lots of them and some are notoriously deep, e.g. 154528 below. I also noticed that at stack 772672 the process has already started existing (cxa_at_exit_wrapper frame). But the log ends at 773152. Did it actually exit, or OOMed during exit? Or you run just subset of tests so that it finishes? A very approx calculation of memory consumption for 773152 stacks of depth 100 gives 613MB: "(773152(32+1008))>>20".

Stack sample for id 154528:
    #0 __libc_init_first ??:? (libc.so.6+0x2dfcf)
    #1 main2() /media/disk2/libo-tsan/sal/cppunittester/cppunittester.cxx:478 (cppunittester+0x4e9f34)
    #2 sal_main() /media/disk2/libo-tsan/sal/cppunittester/cppunittester.cxx:614 (cppunittester+0x4e9f34)
    #3 main /media/disk2/libo-tsan/sal/cppunittester/cppunittester.cxx:609 (cppunittester+0x4e9f34)
    #4 (anonymous namespace)::ProtectedFixtureFunctor::run() const /media/disk2/libo-tsan/sal/cppunittester/cppunittester.cxx:329 (cppunittester+0x4eb248)
    #5 CppUnit::TestRunner::run(CppUnit::TestResult&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestRunner.cpp:96 (libcppunit-1.15.so.1+0x9d721)
    #6 CppUnit::TestResult::runTest(CppUnit::Test*) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestResult.cpp:149 (libcppunit-1.15.so.1+0x947e4)
    #7 CppUnit::TestRunner::WrappingSuite::run(CppUnit::TestResult*) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestRunner.cpp:49 (libcppunit-1.15.so.1+0x9d358)
    #8 CppUnit::TestComposite::run(CppUnit::TestResult*) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestComposite.cpp:23 (libcppunit-1.15.so.1+0x7f785)
    #9 CppUnit::TestComposite::doRunChildTests(CppUnit::TestResult*) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestComposite.cpp:64 (libcppunit-1.15.so.1+0x7fa84)
    #10 CppUnit::TestComposite::run(CppUnit::TestResult*) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestComposite.cpp:23 (libcppunit-1.15.so.1+0x7f785)
    #11 CppUnit::TestComposite::doRunChildTests(CppUnit::TestResult*) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestComposite.cpp:64 (libcppunit-1.15.so.1+0x7fa84)
    #12 CppUnit::TestCase::run(CppUnit::TestResult*) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestCase.cpp:91 (libcppunit-1.15.so.1+0x7e663)
    #13 CppUnit::TestResult::protect(CppUnit::Functor const&, CppUnit::Test*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestResult.cpp:182 (libcppunit-1.15.so.1+0x94bee)
    #14 CppUnit::ProtectorChain::protect(CppUnit::Functor const&, CppUnit::ProtectorContext const&) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/ProtectorChain.cpp:86 (libcppunit-1.15.so.1+0x73d77)
    #15 CppUnit::ProtectorChain::ProtectFunctor::operator()() const /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/ProtectorChain.cpp:20 (libcppunit-1.15.so.1+0x75971)
    #16 CppUnit::DefaultProtector::protect(CppUnit::Functor const&, CppUnit::ProtectorContext const&) /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/DefaultProtector.cpp:15 (libcppunit-1.15.so.1+0x61574)
    #17 CppUnit::ProtectorChain::ProtectFunctor::operator()() const /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/ProtectorChain.cpp:20 (libcppunit-1.15.so.1+0x75971)
    #18 (anonymous namespace)::Prot::protect(CppUnit::Functor const&, CppUnit::ProtectorContext const&) /media/disk2/libo-tsan/unotest/source/cpp/unoexceptionprotector/unoexceptionprotector.cxx:62 (unoexceptionprotector.so+0x2407)
    #19 CppUnit::ProtectorChain::ProtectFunctor::operator()() const /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/ProtectorChain.cpp:20 (libcppunit-1.15.so.1+0x75971)
    #20 (anonymous namespace)::Prot::protect(CppUnit::Functor const&, CppUnit::ProtectorContext const&) /media/disk2/libo-tsan/unotest/source/cpp/unobootstrapprotector/unobootstrapprotector.cxx:78 (unobootstrapprotector.so+0x274e)
    #21 CppUnit::ProtectorChain::ProtectFunctor::operator()() const /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/ProtectorChain.cpp:20 (libcppunit-1.15.so.1+0x75971)
    #22 (anonymous namespace)::Protector::protect(CppUnit::Functor const&, CppUnit::ProtectorContext const&) /media/disk2/libo-tsan/test/source/vclbootstrapprotector.cxx:46 (libvclbootstrapprotector.so+0x134e)
    #23 CppUnit::TestCaseMethodFunctor::operator()() const /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/src/cppunit/TestCase.cpp:32 (libcppunit-1.15.so.1+0x7f00a)
    #24 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560 (libtest_sw_ooxmlexport8.so+0xaeb03)
    #25 CppUnit::TestCaller<testChartProp>::runTest() /media/disk2/libo-tsan/workdir/UnpackedTarball/cppunit/include/cppunit/TestCaller.h:175 (libtest_sw_ooxmlexport8.so+0xaeb03)
    #26 void std::__invoke_impl<void, void (testChartProp::*&)(), testChartProp*&>(std::__invoke_memfun_deref, void (testChartProp::*&)(), testChartProp*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74 (libtest_sw_ooxmlexport8.so+0xaee4a)
    #27 std::__invoke_result<void (testChartProp::*&)(), testChartProp*&>::type std::__invoke<void (testChartProp::*&)(), testChartProp*&>(void (testChartProp::*&)(), testChartProp*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96 (libtest_sw_ooxmlexport8.so+0xaee4a)
    #28 void std::_Bind<void (testChartProp::*(testChartProp*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420 (libtest_sw_ooxmlexport8.so+0xaee4a)
    #29 void std::_Bind<void (testChartProp::*(testChartProp*))()>::operator()<, void>() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503 (libtest_sw_ooxmlexport8.so+0xaee4a)
    #30 void std::__invoke_impl<void, std::_Bind<void (testChartProp::*(testChartProp*))()>&>(std::__invoke_other, std::_Bind<void (testChartProp::*(testChartProp*))()>&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 (libtest_sw_ooxmlexport8.so+0xaee4a)
    #31 std::enable_if<is_invocable_r_v<void, std::_Bind<void (testChartProp::*(testChartProp*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (testChartProp::*(testChartProp*))()>&>(std::_Bind<void (testChartProp::*(testChartProp*))()>&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111 (libtest_sw_ooxmlexport8.so+0xaee4a)
    #32 std::_Function_handler<void (), std::_Bind<void (testChartProp::*(testChartProp*))()> >::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 (libtest_sw_ooxmlexport8.so+0xaee4a)
    #33 testChartProp::Load_Verify_Reload_Verify() /media/disk2/libo-tsan/sw/qa/extras/ooxmlexport/ooxmlexport8.cxx:1199 (libtest_sw_ooxmlexport8.so+0xae90e)
    #34 SwModelTestBase::executeLoadVerifyReloadVerify(char const*, char const*) /media/disk2/libo-tsan/sw/qa/unit/swmodeltestbase.cxx:101 (libswqahelper.so+0xa131)
    #35 SwModelTestBase::load(std::basic_string_view<char16_t, std::char_traits<char16_t> >, char const*, char const*) /media/disk2/libo-tsan/sw/qa/inc/swmodeltestbase.hxx:328 (libswqahelper.so+0x153e4)
    #36 SwModelTestBase::loadURL(rtl::OUString const&, char const*, char const*) /media/disk2/libo-tsan/sw/qa/unit/swmodeltestbase.cxx:513 (libswqahelper.so+0x12a30)
    #37 unotest::MacrosTest::loadFromDesktop(rtl::OUString const&, rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/unotest/source/cpp/macros_test.cxx:63 (libunotest.so+0xd163)
    #38 non-virtual thunk to framework::Desktop::loadComponentFromURL(rtl::OUString const&, rtl::OUString const&, int, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) ??:? (libfwklo.so+0x2477a1)
    #39 framework::Desktop::loadComponentFromURL(rtl::OUString const&, rtl::OUString const&, int, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/framework/source/services/desktop.cxx:593 (libfwklo.so+0x247594)
    #40 framework::LoadEnv::loadComponentFromURL(com::sun::star::uno::Reference<com::sun::star::frame::XComponentLoader> const&, com::sun::star::uno::Reference<com::sun::star::uno::XComponentContext> const&, rtl::OUString const&, rtl::OUString const&, int, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/framework/source/loadenv/loadenv.cxx:168 (libfwklo.so+0x20052b)
    #41 framework::LoadEnv::startLoading(rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::frame::XFrame> const&, rtl::OUString const&, int, LoadEnvFeatures) /media/disk2/libo-tsan/framework/source/loadenv/loadenv.cxx:300 (libfwklo.so+0x201478)
    #42 framework::LoadEnv::start() /media/disk2/libo-tsan/framework/source/loadenv/loadenv.cxx:395 (libfwklo.so+0x20344c)
    #43 framework::LoadEnv::impl_loadContent() /media/disk2/libo-tsan/framework/source/loadenv/loadenv.cxx:1157 (libfwklo.so+0x20742b)
    #44 non-virtual thunk to (anonymous namespace)::SfxFrameLoader_Impl::load(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::frame::XFrame> const&) frmload.cxx:? (libsfxlo.so+0x5820b9)
    #45 (anonymous namespace)::SfxFrameLoader_Impl::load(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::frame::XFrame> const&) /media/disk2/libo-tsan/sfx2/source/view/frmload.cxx:681 (libsfxlo.so+0x57f85b)
    #46 non-virtual thunk to SfxBaseModel::load(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) ??:? (libsfxlo.so+0x4abb95)
    #47 SfxBaseModel::load(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/sfx2/source/doc/sfxbasemodel.cxx:1923 (libsfxlo.so+0x4aadc1)
    #48 SfxObjectShell::DoLoad(SfxMedium*) /media/disk2/libo-tsan/sfx2/source/doc/objstor.cxx:700 (libsfxlo.so+0x462b74)
    #49 SwDocShell::InitNew(com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&) /media/disk2/libo-tsan/sw/source/uibase/app/docshini.cxx:98 (libswlo.so+0x100fead)
    #50 SwDocShell::AddLink() /media/disk2/libo-tsan/sw/source/uibase/app/docshini.cxx:404 (libswlo.so+0x1011085)
    #51 SfxObjectShell::DoLoad(SfxMedium*) /media/disk2/libo-tsan/sfx2/source/doc/objstor.cxx:742 (libsfxlo.so+0x463e45)
    #52 SfxObjectShell::ImportFrom(SfxMedium&, com::sun::star::uno::Reference<com::sun::star::text::XTextRange> const&) /media/disk2/libo-tsan/sfx2/source/doc/objstor.cxx:2262 (libsfxlo.so+0x46fb1f)
    #53 non-virtual thunk to (anonymous namespace)::WriterFilter::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) WriterFilter.cxx:? (libwriterfilterlo.so+0x25c342)
    #54 (anonymous namespace)::WriterFilter::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/writerfilter/source/filter/WriterFilter.cxx:208 (libwriterfilterlo.so+0x2589f3)
    #55 writerfilter::ooxml::OOXMLDocumentImpl::resolve(writerfilter::Stream&) /media/disk2/libo-tsan/writerfilter/source/ooxml/OOXMLDocumentImpl.cxx:504 (libwriterfilterlo.so+0x268d25)
    #56 sax_fastparser::FastSaxParser::parseStream(com::sun::star::xml::sax::InputSource const&) /media/disk2/libo-tsan/sax/source/fastparser/fastparser.cxx:1483 (libexpwraplo.so+0x31c68)
    #57 non-virtual thunk to sax_fastparser::FastSaxParser::parseStream(com::sun::star::xml::sax::InputSource const&) /media/disk2/libo-tsan/sax/source/fastparser/fastparser.cxx:? (libexpwraplo.so+0x31c68)
    #58 sax_fastparser::FastSaxParserImpl::parseStream(com::sun::star::xml::sax::InputSource const&) /media/disk2/libo-tsan/sax/source/fastparser/fastparser.cxx:904 (libexpwraplo.so+0x2a609)
    #59 xmlParseTryOrFinish.isra.0 /build/libxml2-vTtzwD/libxml2-2.9.12+dfsg/builddir/main/../../parser.c:11488 (libxml2.so.2+0x1805e9)
    #60 (anonymous namespace)::call_callbackEndElement(void*, unsigned char const*, unsigned char const*, unsigned char const*) /media/disk2/libo-tsan/sax/source/fastparser/fastparser.cxx:337 (libexpwraplo.so+0x2d325)
    #61 sax_fastparser::FastSaxParserImpl::callbackEndElement() /media/disk2/libo-tsan/sax/source/fastparser/fastparser.cxx:1344 (libexpwraplo.so+0x30932)
    #62 (anonymous namespace)::Entity::endElement() /media/disk2/libo-tsan/sax/source/fastparser/fastparser.cxx:513 (libexpwraplo.so+0x309ef)
    #63 writerfilter::ooxml::OOXMLFastContextHandler::endFastElement(int) /media/disk2/libo-tsan/writerfilter/source/ooxml/OOXMLFastContextHandler.cxx:221 (libwriterfilterlo.so+0x273816)
    #64 non-virtual thunk to writerfilter::ooxml::OOXMLFastContextHandler::endFastElement(int) /media/disk2/libo-tsan/writerfilter/source/ooxml/OOXMLFastContextHandler.cxx:? (libwriterfilterlo.so+0x273816)
    #65 writerfilter::ooxml::OOXMLFastContextHandlerShape::lcl_endFastElement(int) /media/disk2/libo-tsan/writerfilter/source/ooxml/OOXMLFastContextHandler.cxx:1767 (libwriterfilterlo.so+0x27fdba)
    #66 writerfilter::ooxml::OOXMLFastContextHandlerShape::sendShape(int) /media/disk2/libo-tsan/writerfilter/source/ooxml/OOXMLFastContextHandler.cxx:1753 (libwriterfilterlo.so+0x27fbc9)
    #67 writerfilter::LoggedStream::startShape(com::sun::star::uno::Reference<com::sun::star::drawing::XShape> const&) /media/disk2/libo-tsan/writerfilter/source/dmapper/LoggedResources.cxx:138 (libwriterfilterlo.so+0x1af2fa)
    #68 non-virtual thunk to writerfilter::dmapper::DomainMapper::lcl_startShape(com::sun::star::uno::Reference<com::sun::star::drawing::XShape> const&) :? (libwriterfilterlo.so+0xe6812)
    #69 writerfilter::dmapper::DomainMapper::lcl_startShape(com::sun::star::uno::Reference<com::sun::star::drawing::XShape> const&) /media/disk2/libo-tsan/writerfilter/source/dmapper/DomainMapper.cxx:3197 (libwriterfilterlo.so+0xe6759)
    #70 writerfilter::dmapper::DomainMapper_Impl::PushShapeContext(com::sun::star::uno::Reference<com::sun::star::drawing::XShape> const&) /media/disk2/libo-tsan/writerfilter/source/dmapper/DomainMapper_Impl.cxx:3540 (libwriterfilterlo.so+0x1431c5)
    #71 writerfilter::dmapper::DomainMapper_Impl::appendTextContent(com::sun::star::uno::Reference<com::sun::star::text::XTextContent> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/writerfilter/source/dmapper/DomainMapper_Impl.cxx:2530 (libwriterfilterlo.so+0x135bf6)
    #72 SwXText::appendTextContent(com::sun::star::uno::Reference<com::sun::star::text::XTextContent> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/sw/source/core/unocore/unotext.cxx:1509 (libswlo.so+0xe2220c)
    #73 non-virtual thunk to SwXText::appendTextContent(com::sun::star::uno::Reference<com::sun::star::text::XTextContent> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/sw/source/core/unocore/unotext.cxx:? (libswlo.so+0xe2220c)
    #74 SwXText::insertTextContentWithProperties(com::sun::star::uno::Reference<com::sun::star::text::XTextContent> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::text::XTextRange> const&) /media/disk2/libo-tsan/sw/source/core/unocore/unotext.cxx:1472 (libswlo.so+0xe219fd)
    #75 SwXText::insertTextContent(com::sun::star::uno::Reference<com::sun::star::text::XTextRange> const&, com::sun::star::uno::Reference<com::sun::star::text::XTextContent> const&, unsigned char) /media/disk2/libo-tsan/sw/source/core/unocore/unotext.cxx:615 (libswlo.so+0xe18e10)
    #76 non-virtual thunk to SwXFrame::attach(com::sun::star::uno::Reference<com::sun::star::text::XTextRange> const&) :? (libswlo.so+0xbf86c2)
    #77 SwXFrame::attach(com::sun::star::uno::Reference<com::sun::star::text::XTextRange> const&) /media/disk2/libo-tsan/sw/source/core/unocore/unoframe.cxx:3115 (libswlo.so+0xbf8294)
    #78 SwXFrame::attachToRange(com::sun::star::uno::Reference<com::sun::star::text::XTextRange> const&, SwPaM const*) /media/disk2/libo-tsan/sw/source/core/unocore/unoframe.cxx:3068 (libswlo.so+0xbf5be3)
    #79 sw::DocumentContentOperationsManager::InsertEmbObject(SwPaM const&, svt::EmbeddedObjectRef const&, SfxItemSet*) /media/disk2/libo-tsan/sw/source/core/doc/DocumentContentOperationsManager.cxx:3067 (libswlo.so+0x589e1a)
    #80 SwNodes::MakeOLENode(SwNodeIndex const&, svt::EmbeddedObjectRef const&, SwGrfFormatColl*) /media/disk2/libo-tsan/sw/source/core/ole/ndole.cxx:387 (libswlo.so+0x9302a1)
    #81 SwOLENode /media/disk2/libo-tsan/sw/source/core/ole/ndole.cxx:223 (libswlo.so+0x92e74b)
    #82 SwOLEObj::SetNode(SwOLENode*) /media/disk2/libo-tsan/sw/source/core/ole/ndole.cxx:882 (libswlo.so+0x92ea3f)
    #83 comphelper::EmbeddedObjectContainer::InsertEmbeddedObject(com::sun::star::uno::Reference<com::sun::star::embed::XEmbeddedObject> const&, rtl::OUString&) /media/disk2/libo-tsan/comphelper/source/container/embeddedobjectcontainer.cxx:516 (libcomphelper.so+0x7fcef)
    #84 comphelper::EmbeddedObjectContainer::StoreEmbeddedObject(com::sun::star::uno::Reference<com::sun::star::embed::XEmbeddedObject> const&, rtl::OUString&, bool, rtl::OUString const&, rtl::OUString const&) /media/disk2/libo-tsan/comphelper/source/container/embeddedobjectcontainer.cxx:498 (libcomphelper.so+0x7f955)
    #85 non-virtual thunk to OCommonEmbeddedObject::storeAsEntry(com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&, rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) :? (libembobj.so+0x30279)
    #86 OCommonEmbeddedObject::storeAsEntry(com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&, rtl::OUString const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/embeddedobj/source/commonembedding/persistence.cxx:1392 (libembobj.so+0x2fb4d)
    #87 OCommonEmbeddedObject::StoreDocToStorage_Impl(com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, int, rtl::OUString const&, bool) /media/disk2/libo-tsan/embeddedobj/source/commonembedding/persistence.cxx:781 (libembobj.so+0x2a5c5)
    #88 chart::ChartModel::storeToStorage(com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/chart2/source/model/main/ChartModel_Persistence.cxx:764 (libchartcorelo.so+0x1c2d7c)
    #89 non-virtual thunk to chart::ChartModel::storeToStorage(com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/chart2/source/model/main/ChartModel_Persistence.cxx:? (libchartcorelo.so+0x1c2d7c)
    #90 chart::ChartModel::impl_store(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&, com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&) /media/disk2/libo-tsan/chart2/source/model/main/ChartModel_Persistence.cxx:346 (libchartcorelo.so+0x1bcd66)
    #91 non-virtual thunk to chart::XMLFilter::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) :? (libchartcorelo.so+0x186492)
    #92 chart::XMLFilter::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/chart2/source/model/filter/XMLFilter.cxx:213 (libchartcorelo.so+0x181d8f)
    #93 chart::XMLFilter::impl_Export(com::sun::star::uno::Reference<com::sun::star::lang::XComponent> const&, com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/chart2/source/model/filter/XMLFilter.cxx:622 (libchartcorelo.so+0x183633)
    #94 chart::XMLFilter::impl_ExportStream(rtl::OUString const&, rtl::OUString const&, com::sun::star::uno::Reference<com::sun::star::embed::XStorage> const&, com::sun::star::uno::Reference<com::sun::star::xml::sax::XWriter> const&, com::sun::star::uno::Reference<com::sun::star::lang::XMultiServiceFactory> const&, com::sun::star::uno::Sequence<com::sun::star::uno::Any> const&) /media/disk2/libo-tsan/chart2/source/model/filter/XMLFilter.cxx:709 (libchartcorelo.so+0x188051)
    #95 non-virtual thunk to SvXMLExport::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) ??:? (libxolo.so+0x1df6d2)
    #96 SvXMLExport::filter(com::sun::star::uno::Sequence<com::sun::star::beans::PropertyValue> const&) /media/disk2/libo-tsan/xmloff/source/core/xmlexp.cxx:812 (libxolo.so+0x1df0e4)
    #97 SchXMLExport::exportDoc(xmloff::token::XMLTokenEnum) /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:3828 (libxolo.so+0x142626)
    #98 SvXMLExport::exportDoc(xmloff::token::XMLTokenEnum) /media/disk2/libo-tsan/xmloff/source/core/xmlexp.cxx:1377 (libxolo.so+0x1e4510)
    #99 SvXMLExport::ImplExportAutoStyles() /media/disk2/libo-tsan/xmloff/source/core/xmlexp.cxx:1123 (libxolo.so+0x1e3085)
    #100 SchXMLExport::ExportAutoStyles_() /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:3862 (libxolo.so+0x1440a9)
    #101 SchXMLExportHelper_Impl::collectAutoStyles(com::sun::star::uno::Reference<com::sun::star::chart::XChartDocument> const&) /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:1107 (libxolo.so+0x143fe6)
    #102 SchXMLExport::collectAutoStyles() /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:3850 (libxolo.so+0x143fe6)
    #103 SchXMLExportHelper_Impl::parseDocument(com::sun::star::uno::Reference<com::sun::star::chart::XChartDocument> const&, bool, bool) /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:1499 (libxolo.so+0x1238f1)
    #104 SchXMLExportHelper_Impl::exportPlotArea(com::sun::star::uno::Reference<com::sun::star::chart::XDiagram> const&, com::sun::star::uno::Reference<com::sun::star::chart2::XDiagram> const&, com::sun::star::awt::Size const&, bool, bool) /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:2013 (libxolo.so+0x1263a2)
    #105 SchXMLExportHelper_Impl::exportAxes(com::sun::star::uno::Reference<com::sun::star::chart::XDiagram> const&, com::sun::star::uno::Reference<com::sun::star::chart2::XDiagram> const&, bool) /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:2511 (libxolo.so+0x12f981)
    #106 SchXMLExportHelper_Impl::exportAxis(xmloff::token::XMLTokenEnum, xmloff::token::XMLTokenEnum, com::sun::star::uno::Reference<com::sun::star::beans::XPropertySet> const&, com::sun::star::uno::Reference<com::sun::star::chart2::XAxis> const&, rtl::OUString const&, bool, bool, bool, bool, std::basic_string_view<char16_t, std::char_traits<char16_t> >) /media/disk2/libo-tsan/xmloff/source/chart/SchXMLExport.cxx:2369 (libxolo.so+0x137326)
    #107 SvXMLExportPropertyMapper::Filter(SvXMLExport const&, com::sun::star::uno::Reference<com::sun::star::beans::XPropertySet> const&, bool) const /media/disk2/libo-tsan/xmloff/source/style/xmlexppr.cxx:518 (libxolo.so+0x3b9ed7)
    #108 (anonymous namespace)::FilterPropertiesInfo_Impl::FillPropertyStateArray(std::vector<XMLPropertyState, std::allocator<XMLPropertyState> >&, com::sun::star::uno::Reference<com::sun::star::beans::XPropertySet> const&, rtl::Reference<XMLPropertySetMapper> const&, bool) /media/disk2/libo-tsan/xmloff/source/style/xmlexppr.cxx:321 (libxolo.so+0x3bb54a)
    #109 SvXMLExportPropertyMapper::Filter_(SvXMLExport const&, com::sun::star::uno::Reference<com::sun::star::beans::XPropertySet> const&, bool, bool) const /media/disk2/libo-tsan/xmloff/source/style/xmlexppr.cxx:642 (libxolo.so+0x3bb54a)
    #110 non-virtual thunk to chart::WrappedPropertySet::getPropertyStates(com::sun::star::uno::Sequence<rtl::OUString> const&) ??:? (libchartcorelo.so+0x35e4a9)
    #111 chart::WrappedPropertySet::getPropertyStates(com::sun::star::uno::Sequence<rtl::OUString> const&) /media/disk2/libo-tsan/chart2/source/tools/WrappedPropertySet.cxx:317 (libchartcorelo.so+0x35e2f1)
    #112 chart::WrappedPropertySet::getPropertyState(rtl::OUString const&) /media/disk2/libo-tsan/chart2/source/tools/WrappedPropertySet.cxx:286 (libchartcorelo.so+0x35dbe2)
    #113 chart::WrappedProperty::getPropertyState(com::sun::star::uno::Reference<com::sun::star::beans::XPropertyState> const&) const /media/disk2/libo-tsan/chart2/source/tools/WrappedProperty.cxx:105 (libchartcorelo.so+0x35b352)
    #114 chart::wrapper::WrappedScaleProperty::getPropertyValue(com::sun::star::uno::Reference<com::sun::star::beans::XPropertySet> const&) const /media/disk2/libo-tsan/chart2/source/controller/chartapiwrapper/WrappedScaleProperty.cxx:134 (libchartcontrollerlo.so+0x113551)
    #115 chart::wrapper::WrappedScaleProperty::getPropertyValue(chart::wrapper::WrappedScaleProperty::tScaleProperty, com::sun::star::uno::Reference<com::sun::star::beans::XPropertySet> const&) const /media/disk2/libo-tsan/chart2/source/controller/chartapiwrapper/WrappedScaleProperty.cxx:368 (libchartcontrollerlo.so+0x113762)
    #116 chart::wrapper::Chart2ModelContact::getExplicitValuesForAxis(com::sun::star::uno::Reference<com::sun::star::chart2::XAxis> const&, chart::ExplicitScaleData&, chart::ExplicitIncrementData&) /media/disk2/libo-tsan/chart2/source/controller/chartapiwrapper/Chart2ModelContact.cxx:139 (libchartcontrollerlo.so+0xa668e)
    #117 non-virtual thunk to chart::ChartView::getExplicitValuesForAxis(com::sun::star::uno::Reference<com::sun::star::chart2::XAxis>, chart::ExplicitScaleData&, chart::ExplicitIncrementData&) :? (libchartcorelo.so+0x1185b2)
    #118 chart::ChartView::getExplicitValuesForAxis(com::sun::star::uno::Reference<com::sun::star::chart2::XAxis>, chart::ExplicitScaleData&, chart::ExplicitIncrementData&) /media/disk2/libo-tsan/chart2/source/view/main/ChartView.cxx:1670 (libchartcorelo.so+0x117c46)
    #119 chart::ChartView::impl_updateView(bool) /media/disk2/libo-tsan/chart2/source/view/main/ChartView.cxx:2481 (libchartcorelo.so+0x1182fb)
    #120 chart::ChartView::createShapes() /media/disk2/libo-tsan/chart2/source/view/main/ChartView.cxx:2428 (libchartcorelo.so+0x11b2c4)
    #121 chart::ChartView::createShapes2D(com::sun::star::awt::Size const&) /media/disk2/libo-tsan/chart2/source/view/main/ChartView.cxx:2972 (libchartcorelo.so+0x1200f9)
    #122 chart::ChartView::impl_createDiagramAndContent(chart::CreateShapeParam2D const&, com::sun::star::awt::Size const&) /media/disk2/libo-tsan/chart2/source/view/main/ChartView.cxx:1481 (libchartcorelo.so+0x11465c)
    #123 chart::VCoordinateSystem::createMaximumAxesLabels() /media/disk2/libo-tsan/chart2/source/view/axes/VCoordinateSystem.cxx:446 (libchartcorelo.so+0xb0ca0)
    #124 chart::VCartesianAxis::createMaximumLabels() /media/disk2/libo-tsan/chart2/source/view/axes/VCartesianAxis.cxx:1724 (libchartcorelo.so+0xa158d)
    #125 chart::VAxisBase::prepareShapeCreation() /media/disk2/libo-tsan/chart2/source/view/axes/VAxisBase.cxx:179 (libchartcorelo.so+0x938f8)
    #126 chart::VCartesianAxis::createAllTickInfos(std::vector<std::vector<chart::TickInfo, std::allocator<chart::TickInfo> >, std::allocator<std::vector<chart::TickInfo, std::allocator<chart::TickInfo> > > >&) /media/disk2/libo-tsan/chart2/source/view/axes/VCartesianAxis.cxx:673 (libchartcorelo.so+0x9a1d9)
    #127 chart::VAxisBase::createAllTickInfos(std::vector<std::vector<chart::TickInfo, std::allocator<chart::TickInfo> >, std::allocator<std::vector<chart::TickInfo, std::allocator<chart::TickInfo> > > >&) /media/disk2/libo-tsan/chart2/source/view/axes/VAxisBase.cxx:161 (libchartcorelo.so+0x936f5)
    #128 chart::VCartesianAxis::createTickFactory() /media/disk2/libo-tsan/chart2/source/view/axes/VCartesianAxis.cxx:1507 (libchartcorelo.so+0x9fb95)
    #129 chart::VCartesianAxis::createTickFactory2D() /media/disk2/libo-tsan/chart2/source/view/axes/VCartesianAxis.cxx:1514 (libchartcorelo.so+0x9fd03)
    #130 chart::VCartesianAxis::get2DAxisMainLine(basegfx::B2DVector&, basegfx::B2DVector&, chart::AxisLabelAlignment&, double) const /media/disk2/libo-tsan/chart2/source/view/axes/VCartesianAxis.cxx:1437 (libchartcorelo.so+0x9efb8)
    #131 chart::VCartesianAxis::getScreenPosition(double, double, double) const /media/disk2/libo-tsan/chart2/source/view/axes/VCartesianAxis.cxx:1138 (libchartcorelo.so+0x9de48)
    #132 chart::ShapeFactory::createCube(com::sun::star::uno::Reference<com::sun::star::drawing::XShapes> const&, com::sun::star::drawing::Position3D const&, com::sun::star::drawing::Direction3D const&, int, com::sun::star::uno::Reference<com::sun::star::beans::XPropertySet> const&, std::map<rtl::OUString, rtl::OUString, std::less<rtl::OUString>, std::allocator<std::pair<rtl::OUString const, rtl::OUString> > > const&, bool) /media/disk2/libo-tsan/chart2/source/view/main/ShapeFactory.cxx:388 (libchartcorelo.so+0x149c71)
    #133 chart::ShapeFactory::impl_createCube(com::sun::star::uno::Reference<com::sun::star::drawing::XShapes> const&, com::sun::star::drawing::Position3D const&, com::sun::star::drawing::Direction3D const&, int, bool) /media/disk2/libo-tsan/chart2/source/view/main/ShapeFactory.cxx:409 (libchartcorelo.so+0x149fe5)
    #134 Svx3DSceneObject::add(com::sun::star::uno::Reference<com::sun::star::drawing::XShape> const&) /media/disk2/libo-tsan/svx/source/unodraw/unoshap3.cxx:123 (libsvxcorelo.so+0x7040a3)
    #135 SvxShape::Create(SdrObject*, SvxDrawPage*) /media/disk2/libo-tsan/svx/source/unodraw/unoshape.cxx:426 (libsvxcorelo.so+0x71612e)
    #136 SvxShape::setSize(com::sun::star::awt::Size const&) ??:? (libsvxcorelo.so+0x71ae52)
    #137 SvxShape::setSize(com::sun::star::awt::Size const&) /media/disk2/libo-tsan/svx/source/unodraw/unoshape.cxx:1187 (libsvxcorelo.so+0x71ae52)
    #138 SdrObject::GetLogicRect() const /media/disk2/libo-tsan/svx/source/svdraw/svdobj.cxx:1678 (libsvxcorelo.so+0x44dbb1)
    #139 SdrAttrObj::GetSnapRect() const /media/disk2/libo-tsan/svx/source/svdraw/svdoattr.cxx:53 (libsvxcorelo.so+0x444b6c)
    #140 E3dCompoundObject::RecalcSnapRect() /media/disk2/libo-tsan/svx/source/engine3d/obj3d.cxx:555 (libsvxcorelo.so+0x29324e)
    #141 fillViewInformation3DForCompoundObject(drawinglayer::geometry::ViewInformation3D&, E3dCompoundObject const&) /media/disk2/libo-tsan/svx/source/engine3d/helperhittest3d.cxx:138 (libsvxcorelo.so+0x28c5f0)
    #142 sdr::contact::ViewContactOfE3dScene::getViewInformation3D() const /media/disk2/libo-tsan/svx/source/sdr/contact/viewcontactofe3dscene.cxx:355 (libsvxcorelo.so+0x31117c)
    #143 sdr::contact::ViewContactOfE3dScene::getAllContentRange3D() const /media/disk2/libo-tsan/svx/source/sdr/contact/viewcontactofe3dscene.cxx:446 (libsvxcorelo.so+0x3116e9)
    #144 std::deque<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > >::_M_destroy_data(std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:2048 (libsvxcorelo.so+0x28ea3f)
    #145 ~deque /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:1006 (libsvxcorelo.so+0x28ea3f)
    #146 ~Reference /media/disk2/libo-tsan/include/com/sun/star/uno/Reference.hxx:114 (libsvxcorelo.so+0x28ed2a)
    #147 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:140 (libsvxcorelo.so+0x28ed2a)
    #148 void std::_Destroy_aux<false>::__destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152 (libsvxcorelo.so+0x28ed2a)
    #149 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184 (libsvxcorelo.so+0x28ed2a)
    #150 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746 (libsvxcorelo.so+0x28ed2a)
    #151 std::deque<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > >::_M_destroy_data_aux(std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/deque.tcc:872 (libsvxcorelo.so+0x28ed2a)
    #152 cppu::PartialWeakComponentImplHelper<com::sun::star::graphic::XPrimitive3D>::release() /media/disk2/libo-tsan/include/cppuhelper/compbase.hxx:86 (libdrawinglayerlo.so+0xdd819)
    #153 non-virtual thunk to cppu::PartialWeakComponentImplHelper<com::sun::star::graphic::XPrimitive3D>::release() /media/disk2/libo-tsan/include/cppuhelper/compbase.hxx:? (libdrawinglayerlo.so+0xdd819)
    #154 cppu::WeakComponentImplHelperBase::release() /media/disk2/libo-tsan/cppuhelper/source/implbase.cxx:86 (libuno_cppuhelpergcc3.so.3+0x5592e)
    #155 cppu::OWeakObject::release() /media/disk2/libo-tsan/cppuhelper/source/weak.cxx:241 (libuno_cppuhelpergcc3.so.3+0xf99c7)
    #156 ~GroupPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/groupprimitive3d.hxx:49 (libdrawinglayerlo.so+0xfa074)
    #157 ~TransformPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/transformprimitive3d.hxx:40 (libdrawinglayerlo.so+0xfa074)
    #158 ~TransformPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/transformprimitive3d.hxx:40 (libdrawinglayerlo.so+0xfa074)
    #159 std::deque<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > >::_M_destroy_data(std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:2048 (libdrawinglayerlo.so+0x8a8ff)
    #160 ~deque /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:1006 (libdrawinglayerlo.so+0x8a8ff)
    #161 ~Reference /media/disk2/libo-tsan/include/com/sun/star/uno/Reference.hxx:114 (libdrawinglayerlo.so+0x8fa1a)
    #162 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:140 (libdrawinglayerlo.so+0x8fa1a)
    #163 void std::_Destroy_aux<false>::__destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152 (libdrawinglayerlo.so+0x8fa1a)
    #164 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184 (libdrawinglayerlo.so+0x8fa1a)
    #165 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746 (libdrawinglayerlo.so+0x8fa1a)
    #166 std::deque<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > >::_M_destroy_data_aux(std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/deque.tcc:872 (libdrawinglayerlo.so+0x8fa1a)
    #167 cppu::PartialWeakComponentImplHelper<com::sun::star::graphic::XPrimitive3D>::release() /media/disk2/libo-tsan/include/cppuhelper/compbase.hxx:86 (libdrawinglayerlo.so+0xdd819)
    #168 non-virtual thunk to cppu::PartialWeakComponentImplHelper<com::sun::star::graphic::XPrimitive3D>::release() /media/disk2/libo-tsan/include/cppuhelper/compbase.hxx:? (libdrawinglayerlo.so+0xdd819)
    #169 cppu::WeakComponentImplHelperBase::release() /media/disk2/libo-tsan/cppuhelper/source/implbase.cxx:86 (libuno_cppuhelpergcc3.so.3+0x5592e)
    #170 cppu::OWeakObject::release() /media/disk2/libo-tsan/cppuhelper/source/weak.cxx:241 (libuno_cppuhelpergcc3.so.3+0xf99c7)
    #171 ~GroupPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/groupprimitive3d.hxx:49 (libdrawinglayerlo.so+0xfa074)
    #172 ~TransformPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/transformprimitive3d.hxx:40 (libdrawinglayerlo.so+0xfa074)
    #173 ~TransformPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/transformprimitive3d.hxx:40 (libdrawinglayerlo.so+0xfa074)
    #174 std::deque<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > >::_M_destroy_data(std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:2048 (libdrawinglayerlo.so+0x8a8ff)
    #175 ~deque /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:1006 (libdrawinglayerlo.so+0x8a8ff)
    #176 ~Reference /media/disk2/libo-tsan/include/com/sun/star/uno/Reference.hxx:114 (libdrawinglayerlo.so+0x8fa1a)
    #177 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:140 (libdrawinglayerlo.so+0x8fa1a)
    #178 void std::_Destroy_aux<false>::__destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152 (libdrawinglayerlo.so+0x8fa1a)
    #179 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184 (libdrawinglayerlo.so+0x8fa1a)
    #180 void std::_Destroy<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >(com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> >&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746 (libdrawinglayerlo.so+0x8fa1a)
    #181 std::deque<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, std::allocator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D> > >::_M_destroy_data_aux(std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>, std::_Deque_iterator<com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>&, com::sun::star::uno::Reference<com::sun::star::graphic::XPrimitive3D>*>) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/deque.tcc:872 (libdrawinglayerlo.so+0x8fa1a)
    #182 cppu::PartialWeakComponentImplHelper<com::sun::star::graphic::XPrimitive3D>::release() /media/disk2/libo-tsan/include/cppuhelper/compbase.hxx:86 (libdrawinglayerlo.so+0xdd819)
    #183 non-virtual thunk to cppu::PartialWeakComponentImplHelper<com::sun::star::graphic::XPrimitive3D>::release() /media/disk2/libo-tsan/include/cppuhelper/compbase.hxx:? (libdrawinglayerlo.so+0xdd819)
    #184 cppu::WeakComponentImplHelperBase::release() /media/disk2/libo-tsan/cppuhelper/source/implbase.cxx:86 (libuno_cppuhelpergcc3.so.3+0x5592e)
    #185 cppu::OWeakObject::release() /media/disk2/libo-tsan/cppuhelper/source/weak.cxx:241 (libuno_cppuhelpergcc3.so.3+0xf99c7)
    #186 ~GroupPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/groupprimitive3d.hxx:49 (libdrawinglayerlo.so+0xfa07c)
    #187 ~TransformPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/transformprimitive3d.hxx:40 (libdrawinglayerlo.so+0xfa07c)
    #188 ~TransformPrimitive3D /media/disk2/libo-tsan/include/drawinglayer/primitive3d/transformprimitive3d.hxx:40 (libdrawinglayerlo.so+0xfa07c)
    #189 ~Mutex /media/disk2/libo-tsan/include/osl/mutex.hxx:52 (libdrawinglayerlo.so+0xdb9d7)
    #190 ~BaseMutex /media/disk2/libo-tsan/include/cppuhelper/basemutex.hxx:39 (libdrawinglayerlo.so+0xdb9d7)
    #191 ~BasePrimitive3D /media/disk2/libo-tsan/drawinglayer/source/primitive3d/baseprimitive3d.cxx:38 (libdrawinglayerlo.so+0xdb9d7)
    #192 osl_destroyMutex /media/disk2/libo-tsan/sal/osl/unx/mutex.cxx:78 (libuno_sal.so.3+0x5c5ea)
    #193 __interceptor_pthread_mutex_destroy /media/disk2/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1312 (cppunittester+0x47bd73)

[dvyukov]

@vitalybuka I wonder if you stack depot compression/deduplication improvements will help here significantly. In particular it looks like deduping common prefixes should be very profitable here. Do you have any ETA for a prototype available for testing? This may be a good litmus test.

[grandinj]

I also noticed that at stack 772672 the process has already started existing (cxa_at_exit_wrapper frame). But the log ends at 773152. Did it actually exit, or OOMed during exit? Or you run just subset of tests so that it finishes?

I ran a subset of tests so the machine didn't lock up :-) And yes, our tests do run an awful lot of cleanup code And yes, we have abominably deep stack traces

[grandinj]

Is there any sort of pre-filtering that happens on the stacks? Perhaps I can filter some out with judicious use of suppressions?

[dvyukov]

There is no filtering for stacks.

[vitalybuka]

Yes, my "work in progress" very likely will help. I will try to prepare patches next week. I was busy with unrelated suff.

[grandinj]

@vitalybuka any progress on the stack depot compression/ patches?

[vitalybuka]

Yes, please try TSAN_OPTIONS=compress_stack_depot=2 if your tsan build is after https://github.com/llvm/llvm-project/commit/f86deb18cab6479a0961ade3807e4729f3a27bdf

[grandinj]

very very nice @vitalybuka that appears to make a difference of 2 or more orders of magnitude, thanks!

[dvyukov]

Nice! @vitalybuka great work @grandinj thanks for testing