Flaky test in `core-full-stack-tests` - `TileAdmin`

grigasp commented 1 year ago

Build Info

Repo: itwinjs-core
Build Name: iTwin.js
OS: Linux x64
Test Suite Name: core-full-stack-tests
Test Name: TileAdmin tests

Error Detail

  TileAdmin
    ✔ should omit or load edges based on configuration and view flags (352ms)
    ✔ should honor maximum major tile format version (251ms)
##[error]The operation was canceled.

Links to failing builds

https://dev.azure.com/bentleycs/iModelTechnologies/_build/results?buildId=1931673&view=logs&j=0aa84d8c-14fd-5636-fb46-99be2362d8e1&t=234e5986-45f5-5876-92cf-d939451fbda1&l=21317

Notes

It seems the test hang and the pipeline had to be cancelled because of that.

calebmshafer commented 1 year ago

@pmconne @markschlosseratbentley would one of you be able to look into this?

pmconne commented 1 year ago

@calebmshafer display team is doing backlog maintenance later today.

markschlosseratbentley commented 1 year ago

We'll discuss later today, yes. Sounds good.

markschlosseratbentley commented 1 year ago

@markschlosseratbentley will try to repro locally.

markschlosseratbentley commented 1 year ago

So far cannot reproduce locally on Linux machine. @grigasp is there any way to tell how often this has happened?

grigasp commented 1 year ago

So far cannot reproduce locally on Linux machine. @grigasp is there any way to tell how often this has happened?

I don't have an automated way to find these failures - just open failed iTwin.js pipeline runs of the last few days and see why they failed. From what I saw there were a few core-full-stack-test failures a day lately, most of them crashes.

Also, I'm not sure if this is really TileAdmin - similar crashes showed up on a number of different tests, TileAdmin tests being one of the more frequent spots.

markschlosseratbentley commented 1 year ago

FYI I ran rush cover in a loop on my Linux laptop, based off iTwin.js master. It eventually hung -- but not on TileAdmin stuff. It hung on presentation tests. I have a call stack by attaching gdb to node, but I did not locally build the backend for this experiment, meaning symbols aren't great (?? in the callstack in places).

cc @MichaelBelousov

Hung here:

==[ presentation-full-stack-tests ]==============================[ 22 of 78 ]==

Partial callstack attaching gdb:

gdb backtrace

(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f49222dae79 in folly::detail::Futex::futexWaitImpl(unsigned int, std::chrono::time_point > >*, std::chrono::time_point > >*, unsigned int) ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#2  0x00007f49221e5bfd in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#3  0x00007f49221e5971 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#4  0x00007f49221e590b in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#5  0x00007f49228542af in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#6  0x00007f492282c6f1 in BentleyM0200::ECPresentation::ConnectionManager::BroadcastEvent(BentleyM0200::ECPresentation::ConnectionEvent const&) const ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f492282b812 in BentleyM0200::ECPresentation::ConnectionManager::NotifyConnectionClosed(BentleyM0200::Utf8String const&) ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f492282b6fc in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f492282fed0 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#10 0x00007f492282ff17 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#11 0x00007f4922132099 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modul--Type  for more, q to quit, c to continue without paging--
es/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#12 0x00007f492247f876 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#13 0x00007f492247f83a in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#14 0x00007f49224788d5 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#15 0x00007f492247890b in BentleyM0200::BeSQLite::Db::CloseDb() ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#16 0x00007f492216e6b8 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#17 0x00007f492216eda8 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#18 0x00007f492216ebba in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#19 0x0000000000ab45dd in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo const&) ()
#20 0x0000000000d55fae in v8::internal::MaybeHandle v8::internal::(anonymous namespace)::HandleApiCallHelper(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::BuiltinArguments) ()
#21 0x0000000000d573cf in v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) ()
#22 0x00000000015f2e19 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
#23 0x00007f4bbf2593b1 in ?? ()
#24 0x00003932b76c1599 in ?? ()
#25 0x00002c794e6f6621 in ?? ()
#26 0x0000000500000000 in ?? ()
#27 0x00003932b76c1669 in ?? ()
#28 0x00003790a3229549 in ?? ()
#29 0x00003790a32091d1 in ?? ()
--Type  for more, q to quit, c to continue without paging--
#30 0x00003790a3229549 in ?? ()
#31 0x00002c794e6f6621 in ?? ()
#32 0x00003932b76c1599 in ?? ()
#33 0x00003790a3229561 in ?? ()
#34 0x000014b037be61a1 in ?? ()
#35 0x000036e041c727f9 in ?? ()
#36 0x0000000000000000 in ?? ()

grigasp commented 1 year ago

Filed an issue for the hang in presentation-full-stack-tests. However, I'm convinced it's not the reason core-full-stack-tests fail so often in CI.

MichaelBelousov commented 1 year ago

@markschlosseratbentley no other interesting threads? Also can you use <details><summary>thread 1</summary>(copy backtrace here)</details> to prevent unnecesarily long messages?

MichaelBelousov commented 1 year ago

also @markschlosseratbentley if you pull the commit for that release of the addon and build it debug on linux, you might be able to get gdb to load those symbols in right now

markschlosseratbentley commented 1 year ago

gdb backtrace with more symbols

(gdb) add-symbol-file /home/markschlosser/temp/imodeljs-linux-x64/imodeljs.node
add symbol table from file "/home/markschlosser/temp/imodeljs-linux-x64/imodeljs.node"
(y or n) y
Reading symbols from /home/markschlosser/temp/imodeljs-linux-x64/imodeljs.node...
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f49222dae79 in folly::detail::Futex::futexWaitImpl(unsigned int, std::chrono::time_point > >*, std::chrono::time_point > >*, unsigned int) ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#2  0x00007f49221e5bfd in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#3  0x00007f49221e5971 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#4  0x00007f49221e590b in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#5  0x00007f49228542af in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#6  0x00007f492282c6f1 in BentleyM0200::ECPresentation::ConnectionManager::BroadcastEvent(BentleyM0200::ECPresentation::ConnectionEvent const&) const ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f492282b812 in BentleyM0200::ECPresentation::ConnectionManager::NotifyConnectionClosed(BentleyM0200::Utf8String const&) ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f492282b6fc in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f492282fed0 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#10 0x00007f492282ff17 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#11 0x00007f4922132099 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modul--Type  for more, q to quit, c to continue without paging-- 
es/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#12 0x00007f492247f876 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#13 0x00007f492247f83a in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#14 0x00007f49224788d5 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#15 0x00007f492247890b in BentleyM0200::BeSQLite::Db::CloseDb() ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#16 0x00007f492216e6b8 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#17 0x00007f492216eda8 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#18 0x00007f492216ebba in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#19 0x0000000000ab45dd in std::isinf (__x=0) at /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/cmath:593
#20 0x0000000000d55fae in BentleyM0200::PolyfaceVisitor::ClearAllArrays (this=0x7fff59ee9940)
    at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/GeomLibs/common/inc/../../geom/src/polyface/PolyfaceVisitor.cpp:272
#21 0x0000000000d573cf in BentleyM0200::PolyfaceVisitor::PushInterpolatedFaceData (this=0x7f4bbf296e43, source=..., 
    i0=140734702197064, fraction=3.107178276641521e-310, i1=36305445403481, suppressVisibility=false)
    at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/GeomLibs/common/inc/../../geom/src/polyface/PolyfaceVisitor.cpp:930
#22 0x00000000015f2e19 in BentleyM0200::Dgn::Centerline::Output (this=0xffffffff00000000, context=..., 
    pStroke=0x3932b76c1669, normal=0x3e5e56f37819, startTangent=0x3932b76c1599, endTangent=0x7fff59ee9940)
    at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/iModelPlatform/DgnCore/linestyle/StrokePattern.cpp:1811
#23 0x00007f4bbf2593b1 in ?? ()
#24 0x00003932b76c1599 in ?? ()
#25 0x00002c794e6f6621 in ?? ()
--Type  for more, q to quit, c to continue without paging--
#26 0x0000000500000000 in ?? ()
#27 0x00003932b76c1669 in ?? ()
#28 0x00003790a3229549 in ?? ()
#29 0x00003790a32091d1 in ?? ()
#30 0x00003790a3229549 in ?? ()
#31 0x00002c794e6f6621 in ?? ()
#32 0x00003932b76c1599 in ?? ()
#33 0x00003790a3229561 in ?? ()
#34 0x000014b037be61a1 in ?? ()
#35 0x000036e041c727f9 in ?? ()
#36 0x0000000000000000 in ?? ()

markschlosseratbentley commented 1 year ago

gdb backtrace with ndebug symbols

Reading symbols from /home/markschlosser/temp/ndebug/imodeljs-linux-x64/imodeljs.node...
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f49222dae79 in folly::detail::Futex::futexWaitImpl(unsigned int, std::chrono::time_point > >*, std::chrono::time_point > >*, unsigned int) ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#2  0x00007f49221e5bfd in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#3  0x00007f49221e5971 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#4  0x00007f49221e590b in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#5  0x00007f49228542af in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#6  0x00007f492282c6f1 in BentleyM0200::ECPresentation::ConnectionManager::BroadcastEvent(BentleyM0200::ECPresentation::ConnectionEvent const&) const ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f492282b812 in BentleyM0200::ECPresentation::ConnectionManager::NotifyConnectionClosed(BentleyM0200::Utf8String const&) ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f492282b6fc in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f492282fed0 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#10 0x00007f492282ff17 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#11 0x00007f4922132099 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modul--Type  for more, q to quit, c to continue without paging--
es/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#12 0x00007f492247f876 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#13 0x00007f492247f83a in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#14 0x00007f49224788d5 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#15 0x00007f492247890b in BentleyM0200::BeSQLite::Db::CloseDb() ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#16 0x00007f492216e6b8 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#17 0x00007f492216eda8 in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#18 0x00007f492216ebba in ?? ()
   from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#19 0x0000000000ab45dd in BentleyM0200::Dgn::GeometryBuilder::OnNewGeom (this=0x61999c0, localRangeIn=..., 
    isSubGraphic=255, opCode=BentleyM0200::Dgn::GeometryStreamIO::OpCode::Invalid)
    at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/iModelPlatform/DgnCore/ElementGeometry.cpp:5753
#20 0x0000000000d55fae in BentleyM0200::RefCountedPtr::~RefCountedPtr (this=)
    at /home/mike/work/itwinjs-cospace/repos/imodel02/out/LinuxX64/static/BuildContexts/ECPresentation/PublicAPI/Bentley/RefCounted.h:138
#21 BentleyM0200::ECPresentation::QueryBuilderHelpers::AddToUnionSet (set=..., query=...)
    at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ECPresentation/Source/Shared/Queries/QueryBuilderHelpers.cpp:35
#22 0x0000000000d573cf in BentleyM0200::ECPresentation::RecursiveQueriesHelper::RecursivelySelectRelatedKeys (
    result=std::unordered_map with 1255660970620028165 elements, connection=..., partialQuery=..., sourceIds=...)
    at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ECPresentation/Source/Shared/Querie--Type  for more, q to quit, c to continue without paging--
s/QueryBuilderHelpers.cpp:329
#23 0x00000000015f2e19 in clash_segs ()
#24 0x00003e5e56f347a9 in ?? ()
#25 0x0000000000000001 in ?? ()
#26 0x0000000000000001 in ?? ()
#27 0x0000328827b01139 in ?? ()
#28 0x0000000000000004 in ?? ()
#29 0x0000328827b01139 in ?? ()
#30 0x0000000005cbad10 in GCC_except_table497 ()
#31 0x000000000000001c in ?? ()
#32 0x00007fff59eea0c0 in ?? ()
#33 0x0000000001583943 in BLE__valid_overflows ()
#34 0x00007fff59eeb416 in ?? ()
#35 0x00007fff59eeb478 in ?? ()
#36 0x00007fff59eeb4bd in ?? ()
#37 0x00007fff59eeb4dd in ?? ()
#38 0x00007fff59eeb4f9 in ?? ()
#39 0x00007fff59eeb52a in ?? ()
#40 0x00007fff59eeb586 in ?? ()
#41 0x00007fff59eeb59c in ?? ()
#42 0x00007fff59eeb5d2 in ?? ()
#43 0x00007fff59eeb5e6 in ?? ()
#44 0x00007fff59eeb5ee in ?? ()
#45 0x00007fff59eeb60f in ?? ()
#46 0x00007fff59eeb647 in ?? ()
#47 0x00007fff59eeb68e in ?? ()
#48 0x00007fff59eeb6a0 in ?? ()
#49 0x00007fff59eeb6ba in ?? ()
#50 0x00007fff59eeb6d0 in ?? ()
#51 0x00007fff59eeb6e6 in ?? ()
#52 0x00007fff59eeb728 in ?? ()
#53 0x00007fff59eeb73f in ?? ()
#54 0x00007fff59eeb756 in ?? ()
#55 0x00007fff59eeb76a in ?? ()
#56 0x00007fff59eeb794 in ?? ()
#57 0x00007fff59eeb7c0 in ?? ()
#58 0x00007fff59eeb7ec in ?? ()
--Type  for more, q to quit, c to continue without paging--
#59 0x00007fff59eeb86b in ?? ()
#60 0x00007fff59eebbe5 in ?? ()
#61 0x00007fff59eebc3f in ?? ()
#62 0x00007fff59eebc7d in ?? ()
#63 0x00007fff59eebcac in ?? ()
#64 0x00007fff59eebcc3 in ?? ()
#65 0x00007fff59eebd19 in ?? ()
#66 0x00007fff59eebd30 in ?? ()
#67 0x00007fff59eebd4f in ?? ()
#68 0x00007fff59eebd5a in ?? ()
#69 0x00007fff59eebd6b in ?? ()
#70 0x00007fff59eebd8c in ?? ()
#71 0x00007fff59eebddc in ?? ()
#72 0x00007fff59eebdf0 in ?? ()
#73 0x00007fff59eebe0b in ?? ()
#74 0x00007fff59eebe41 in ?? ()
#75 0x00007fff59eec430 in ?? ()
#76 0x00007fff59eec44e in ?? ()
#77 0x00007fff59eec469 in ?? ()
#78 0x00007fff59eec4dd in ?? ()
#79 0x00007fff59eec4fe in ?? ()
#80 0x00007fff59eec527 in ?? ()
#81 0x00007fff59eec537 in ?? ()
#82 0x00007fff59eec571 in ?? ()
#83 0x00007fff59eec58b in ?? ()
#84 0x00007fff59eec5a4 in ?? ()
#85 0x00007fff59eeccc6 in ?? ()
#86 0x00007fff59eeccd9 in ?? ()
#87 0x00007fff59eecceb in ?? ()
#88 0x00007fff59eecd0d in ?? ()
#89 0x00007fff59eecd1f in ?? ()
#90 0x00007fff59eecd71 in ?? ()
#91 0x00007fff59eecda9 in ?? ()
#92 0x00007fff59eecdf1 in ?? ()
#93 0x00007fff59eece52 in ?? ()
#94 0x00007fff59eece7f in ?? ()
#95 0x00007fff59eece8d in ?? ()
--Type  for more, q to quit, c to continue without paging--
#96 0x00007fff59eecee2 in ?? ()
#97 0x00007fff59eecf2b in ?? ()
#98 0x00007fff59eecf42 in ?? ()
#99 0x00007fff59eecf53 in ?? ()
#100 0x00007fff59eecf68 in ?? ()
#101 0x00007fff59eecfb5 in ?? ()
#102 0x0000000000000000 in ?? ()

markschlosseratbentley commented 1 year ago

gdb backtrace with ndebug symbols for all threads

(gdb) thread apply all bt

Thread 28 (Thread 0x7f4bbc669640 (LWP 54665) "node"):
#0  futex_wait (private=0, expected=2, futex_word=0x7f48c823c248) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7f48c823c248, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f4bcd8c20dd in lll_mutex_lock_optimized (mutex=0x7f48c823c248) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7f48c823c248) at ./nptl/pthread_mutex_lock.c:128
#4  0x00007f4922300186 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#5  0x00007f492230022f in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#6  0x00007f49223001b8 in BentleyM0200::BeMutexHolder::BeMutexHolder(BentleyM0200::BeMutex&, BentleyM0200::BeMutexHolder::Lock) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49228d097e in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49228d0a38 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f49228d0a38 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#10 0x00007f49228d0a38 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#11 0x00007f49228d0a38 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#12 0x00007f49228d0a38 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#13 0x00007f49228d1226 in BentleyM0200::ECPresentation::Diagnostics::AddValueToArrayAttribute(char const*, BentleyM0200::Utf8String, bool) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#14 0x00007f49228d3656 in BentleyM0200::ECPresentation::DiagnosticsHelpers::ReportRule(BentleyM0200::ECPresentation::PresentationKey const&) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#15 0x00007f49254bd373 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#16 0x00007f4926ac2d4b in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#17 0x00007f49254be71f in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#18 0x00007f49254c026c in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#19 0x00007f49254c1d9b in BentleyM0200::ECPresentation::ContentSpecificationsHandler::HandleSpecification(BentleyM0200::ECPresentation::SelectedNodeInstancesSpecification const&, BentleyM0200::ECPresentation::IParsedInput const&) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#20 0x00007f4926ac0d41 in BentleyM0200::ECPresentation::ContentDescriptorBuilder::CreateDescriptor(BentleyM0200::ECPresentation::SelectedNodeInstancesSpecification const&, BentleyM0200::ECPresentation::IParsedInput const&) const () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#21 0x00007f49254a4f9f in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imo--Type  for more, q to quit, c to continue without paging--
deljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#22 0x00007f4925490160 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#23 0x00007f492549f2bf in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#24 0x00007f492549ec19 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#25 0x00007f492289c1f0 in BentleyM0200::ECPresentation::RulesDrivenECPresentationManagerImpl::_GetContentDescriptor(BentleyM0200::ECPresentation::ImplTaskParams const&) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#26 0x00007f4922877154 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#27 0x00007f4922869cab in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#28 0x00007f4922869960 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#29 0x00007f492295ed7f in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#30 0x00007f492219f353 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#31 0x00007f49222cf9d6 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#32 0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#33 0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#34 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 27 (Thread 0x7f491bdfe640 (LWP 52519) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d8c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/commo--Type  for more, q to quit, c to continue without paging--
n/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 26 (Thread 0x7f491bfff640 (LWP 52518) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d8c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 25 (Thread 0x7f49203f3640 (LWP 52517) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
--Type  for more, q to quit, c to continue without paging--
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 24 (Thread 0x7f49205f4640 (LWP 52516) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 23 (Thread 0x7f49207f5640 (LWP 52515) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
--Type  for more, q to quit, c to continue without paging--

Thread 22 (Thread 0x7f49209f6640 (LWP 52514) "node"):
#0  futex_wait (private=0, expected=2, futex_word=0x7f48c824ba58) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7f48c824ba58, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f4bcd8c20dd in lll_mutex_lock_optimized (mutex=0x7f48c824ba58) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7f48c824ba58) at ./nptl/pthread_mutex_lock.c:128
#4  0x00007f4922300186 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#5  0x00007f492230022f in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#6  0x00007f49223001b8 in BentleyM0200::BeMutexHolder::BeMutexHolder(BentleyM0200::BeMutex&, BentleyM0200::BeMutexHolder::Lock) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49228cffce in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49228d0060 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f49228d0060 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#10 0x00007f49228d0060 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#11 0x00007f49228d0060 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#12 0x00007f49221a1a50 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#13 0x00007f49221a19a0 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#14 0x00007f49221a15b8 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#15 0x00007f49221a1532 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#16 0x00007f49221a1467 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#17 0x00007f492219f353 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#18 0x00007f49222cf9d6 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#19 0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#20 0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#21 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 21 (Thread 0x7f4920bf7640 (LWP 52512) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at--Type  for more, q to quit, c to continue without paging--
 ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 20 (Thread 0x7f4920df8640 (LWP 52511) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 19 (Thread 0x7f4920ff9640 (LWP 52510) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid--Type  for more, q to quit, c to continue without paging--
=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 18 (Thread 0x7f49211fa640 (LWP 52509) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 17 (Thread 0x7f49213fb640 (LWP 52508) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./np--Type  for more, q to quit, c to continue without paging--
tl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 16 (Thread 0x7f49215fc640 (LWP 52507) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 15 (Thread 0x7f49217fd640 (LWP 52506) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d8c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
--Type  for more, q to quit, c to continue without paging--
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 14 (Thread 0x7f49219fe640 (LWP 52505) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 13 (Thread 0x7f4921bff640 (LWP 52504) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d8c) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d8c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
--Type  for more, q to quit, c to continue without paging--
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 12 (Thread 0x7f4bbc267640 (LWP 52503) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x6635d88) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x6635d88, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x6635de0, cond=0x6635d60) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x6635d60, mutex=0x6635de0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f4bcdc0c77d in std::condition_variable::wait(std::unique_lock&) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f4922300389 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f49222cf9a1 in BeFolly::ThreadPool::Worker::Work() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f49222cfb5a in BeFolly::ThreadPool::Worker::Main(void*) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#10 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 11 (Thread 0x7f4bbce6a640 (LWP 52498) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x4a44c48 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x4a44be0 , cond=0x4a44c20 ) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ../deps/uv/src/unix/thread.c:780
--Type  for more, q to quit, c to continue without paging--
#6  0x00000000015607a4 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#7  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#8  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7f4bbd66b640 (LWP 52497) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x4a44c48 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x4a44be0 , cond=0x4a44c20 ) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ../deps/uv/src/unix/thread.c:780
#6  0x00000000015607a4 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#7  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#8  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7f4bbde6c640 (LWP 52496) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x4a44c48 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x4a44be0 , cond=0x4a44c20 ) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ../deps/uv/src/unix/thread.c:780
#6  0x00000000015607a4 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#7  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#8  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7f4bbe66d640 (LWP 52495) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x4a44c48 ) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x4a44c48 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
--Type  for more, q to quit, c to continue without paging--
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x4a44be0 , cond=0x4a44c20 ) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=cond@entry=0x4a44c20 , mutex=mutex@entry=0x4a44be0 ) at ../deps/uv/src/unix/thread.c:780
#6  0x00000000015607a4 in worker (arg=0x0) at ../deps/uv/src/threadpool.c:76
#7  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#8  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7f4bcdda9640 (LWP 51876) "node"):
#0  __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x4a391a0 ) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=, abstime=0x0, clockid=0, expected=0, futex_word=0x4a391a0 ) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x4a391a0 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8c6c5f in do_futex_wait (sem=sem@entry=0x4a391a0 , abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111
#4  0x00007f4bcd8c6cf8 in __new_sem_wait_slow64 (sem=sem@entry=0x4a391a0 , abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183
#5  0x00007f4bcd8c6d71 in __new_sem_wait (sem=sem@entry=0x4a391a0 ) at ./nptl/sem_wait.c:42
#6  0x00000000015742c2 in uv__sem_wait (sem=) at ../deps/uv/src/unix/thread.c:626
#7  uv_sem_wait (sem=0x4a391a0 ) at ../deps/uv/src/unix/thread.c:682
#8  0x0000000000c15215 in BentleyM0200::RefCountedPtr::~RefCountedPtr (this=) at /home/mike/work/itwinjs-cospace/repos/imodel02/out/LinuxX64/static/BuildContexts/ECObjects/PublicAPI/Bentley/RefCounted.h:138
#9  BentleyM0200::BeJsConst::~BeJsConst (this=) at /home/mike/work/itwinjs-cospace/repos/imodel02/out/LinuxX64/static/BuildContexts/ECObjects/PublicAPI/json/BeJsValue.h:133
#10 BentleyM0200::ECN::PropertyCategory::ToJson (this=0x7f4bcdda9640, outValue=..., standalone=, includeSchemaVersion=) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ecobjects/src/PropertyCategory.cpp:138
#11 0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#12 0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f4bc77fe640 (LWP 51874) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5cddf20) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5cddf20) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5cddf20, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5cdded0, cond=0x5cddef8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5cddef8, mutex=0x5cdded0) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=, mutex=) at ../deps/uv/src/unix/thread.c:780
--Type  for more, q to quit, c to continue without paging--
#6  0x0000000000b7716b in __gnu_cxx::__normal_iterator > >::operator++ (this=) at /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_iterator.h:1107
#7  BentleyM0200::ECN::ECClass::ConvertPropertyToPrimitiveArray (ecClass=0x0, startingClass=0x5cddef8, propName=, includeDerivedClasses=) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ecobjects/src/ECClass.cpp:1015
#8  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#9  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f4bc7fff640 (LWP 51873) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5cddf24) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5cddf24) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5cddf24, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5cdded0, cond=0x5cddef8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5cddef8, mutex=0x5cdded0) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=, mutex=) at ../deps/uv/src/unix/thread.c:780
#6  0x0000000000b7716b in __gnu_cxx::__normal_iterator > >::operator++ (this=) at /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_iterator.h:1107
#7  BentleyM0200::ECN::ECClass::ConvertPropertyToPrimitiveArray (ecClass=0x0, startingClass=0x5cddef8, propName=, includeDerivedClasses=) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ecobjects/src/ECClass.cpp:1015
#8  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#9  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f4bcc824640 (LWP 51872) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5cddf24) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5cddf24) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5cddf24, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5cdded0, cond=0x5cddef8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5cddef8, mutex=0x5cdded0) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=, mutex=) at ../deps/uv/src/unix/thread.c:780
#6  0x0000000000b7716b in __gnu_cxx::__normal_iterator > >::operator++ (this=) at /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_iterator.h:1107
#7  BentleyM0200::ECN::ECClass::ConvertPropertyToPrimitiveArray (ecClass=0x0, startingClass=0x5cddef8, propName=, includeDerivedClasses=) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ecobjects/src/ECClass.cpp:1015
--Type  for more, q to quit, c to continue without paging--
#8  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#9  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f4bcd025640 (LWP 51871) "node"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5cddf20) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5cddf20) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5cddf20, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f4bcd8bdac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5cdded0, cond=0x5cddef8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5cddef8, mutex=0x5cdded0) at ./nptl/pthread_cond_wait.c:627
#5  0x0000000001574489 in uv_cond_wait (cond=, mutex=) at ../deps/uv/src/unix/thread.c:780
#6  0x0000000000b7716b in __gnu_cxx::__normal_iterator > >::operator++ (this=) at /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_iterator.h:1107
#7  BentleyM0200::ECN::ECClass::ConvertPropertyToPrimitiveArray (ecClass=0x0, startingClass=0x5cddef8, propName=, includeDerivedClasses=) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ecobjects/src/ECClass.cpp:1015
#8  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#9  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f4bcd826640 (LWP 51870) "node"):
#0  0x00007f4bcd94ffde in epoll_wait (epfd=9, events=events@entry=0x7f4bcd822ce0, maxevents=maxevents@entry=1024, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x0000000001577eb7 in uv__io_poll (loop=loop@entry=0x5ce2368, timeout=) at ../deps/uv/src/unix/epoll.c:236
#2  0x0000000001565f38 in uv_run (loop=0x5ce2368, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:389
#3  0x0000000000b7c2ab in BentleyM0200::ECN::ECEntityClass::_ToJson (this=, outValue=..., standalone=215, includeSchemaVersion=56, includeInheritedProperties=) at /home/mike/work/itwinjs-cospace/repos/imodel02/out/LinuxX64/static/BuildContexts/ECObjects/PublicAPI/Bentley/RefCounted.h:141
#4  0x00007f4bcd8beb43 in start_thread (arg=) at ./nptl/pthread_create.c:442
#5  0x00007f4bcd950a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f4bcd8277c0 (LWP 51858) "node"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f49222dae79 in folly::detail::Futex::futexWaitImpl(unsigned int, std::chrono::time_point > >*, std::chrono::time_point > >*, unsigned int) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#2  0x00007f49221e5bfd in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#3  0x00007f49221e5971 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#4  0x00007f49221e590b in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imo--Type  for more, q to quit, c to continue without paging--
deljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#5  0x00007f49228542af in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#6  0x00007f492282c6f1 in BentleyM0200::ECPresentation::ConnectionManager::BroadcastEvent(BentleyM0200::ECPresentation::ConnectionEvent const&) const () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#7  0x00007f492282b812 in BentleyM0200::ECPresentation::ConnectionManager::NotifyConnectionClosed(BentleyM0200::Utf8String const&) () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8  0x00007f492282b6fc in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9  0x00007f492282fed0 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#10 0x00007f492282ff17 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#11 0x00007f4922132099 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#12 0x00007f492247f876 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#13 0x00007f492247f83a in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#14 0x00007f49224788d5 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#15 0x00007f492247890b in BentleyM0200::BeSQLite::Db::CloseDb() () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#16 0x00007f492216e6b8 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#17 0x00007f492216eda8 in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#18 0x00007f492216ebba in ?? () from /home/markschlosser/code/itwinjs-core/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.6.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#19 0x0000000000ab45dd in BentleyM0200::Dgn::GeometryBuilder::OnNewGeom (this=0x61999c0, localRangeIn=..., isSubGraphic=255, opCode=BentleyM0200::Dgn::GeometryStreamIO::OpCode::Invalid) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/iModelPlatform/DgnCore/ElementGeometry.cpp:5753
#20 0x0000000000d55fae in BentleyM0200::RefCountedPtr::~RefCountedPtr (this=) at /home/mike/work/itwinjs-cospace/repos/imodel02/out/LinuxX64/static/BuildContexts/ECPresentation/PublicAPI/Bentley/RefCounted.h:138
#21 BentleyM0200::ECPresentation::QueryBuilderHelpers::AddToUnionSet (set=..., query=...) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ECPresentation/Source/Shared/Queries/QueryBuilderHelpers.cpp:35
#22 0x0000000000d573cf in BentleyM0200::ECPresentation::RecursiveQueriesHelper::RecursivelySelectRelatedKeys (result=std::unordered_map with 1255660970620028165 elements, connection=..., partialQuery=..., sourceIds=...) at /home/mike/work/itwinjs-cospace/repos/imodel02/src/imodel-native/iModelCore/ECPresentation/Source/Shared/Queries/QueryBuilderHelpers.cpp:329
#23 0x00000000015f2e19 in clash_segs ()
#24 0x00003e5e56f347a9 in ?? ()
#25 0x0000000000000001 in ?? ()
#26 0x0000000000000001 in ?? ()
--Type  for more, q to quit, c to continue without paging--
#27 0x0000328827b01139 in ?? ()
#28 0x0000000000000004 in ?? ()
#29 0x0000328827b01139 in ?? ()
#30 0x0000000005cbad10 in GCC_except_table497 ()
#31 0x000000000000001c in ?? ()
#32 0x00007fff59eea0c0 in ?? ()
#33 0x0000000001583943 in BLE__valid_overflows ()
#34 0x00007fff59eeb416 in ?? ()
#35 0x00007fff59eeb478 in ?? ()
#36 0x00007fff59eeb4bd in ?? ()
#37 0x00007fff59eeb4dd in ?? ()
#38 0x00007fff59eeb4f9 in ?? ()
#39 0x00007fff59eeb52a in ?? ()
#40 0x00007fff59eeb586 in ?? ()
#41 0x00007fff59eeb59c in ?? ()
#42 0x00007fff59eeb5d2 in ?? ()
#43 0x00007fff59eeb5e6 in ?? ()
#44 0x00007fff59eeb5ee in ?? ()
#45 0x00007fff59eeb60f in ?? ()
#46 0x00007fff59eeb647 in ?? ()
#47 0x00007fff59eeb68e in ?? ()
#48 0x00007fff59eeb6a0 in ?? ()
#49 0x00007fff59eeb6ba in ?? ()
#50 0x00007fff59eeb6d0 in ?? ()
#51 0x00007fff59eeb6e6 in ?? ()
#52 0x00007fff59eeb728 in ?? ()
#53 0x00007fff59eeb73f in ?? ()
#54 0x00007fff59eeb756 in ?? ()
#55 0x00007fff59eeb76a in ?? ()
#56 0x00007fff59eeb794 in ?? ()
#57 0x00007fff59eeb7c0 in ?? ()
#58 0x00007fff59eeb7ec in ?? ()
#59 0x00007fff59eeb86b in ?? ()
#60 0x00007fff59eebbe5 in ?? ()
#61 0x00007fff59eebc3f in ?? ()
#62 0x00007fff59eebc7d in ?? ()
#63 0x00007fff59eebcac in ?? ()
#64 0x00007fff59eebcc3 in ?? ()
#65 0x00007fff59eebd19 in ?? ()
#66 0x00007fff59eebd30 in ?? ()
#67 0x00007fff59eebd4f in ?? ()
#68 0x00007fff59eebd5a in ?? ()
#69 0x00007fff59eebd6b in ?? ()
#70 0x00007fff59eebd8c in ?? ()
#71 0x00007fff59eebddc in ?? ()
#72 0x00007fff59eebdf0 in ?? ()
#73 0x00007fff59eebe0b in ?? ()
#74 0x00007fff59eebe41 in ?? ()
--Type  for more, q to quit, c to continue without paging--
#75 0x00007fff59eec430 in ?? ()
#76 0x00007fff59eec44e in ?? ()
#77 0x00007fff59eec469 in ?? ()
#78 0x00007fff59eec4dd in ?? ()
#79 0x00007fff59eec4fe in ?? ()
#80 0x00007fff59eec527 in ?? ()
#81 0x00007fff59eec537 in ?? ()
#82 0x00007fff59eec571 in ?? ()
#83 0x00007fff59eec58b in ?? ()
#84 0x00007fff59eec5a4 in ?? ()
#85 0x00007fff59eeccc6 in ?? ()
#86 0x00007fff59eeccd9 in ?? ()
#87 0x00007fff59eecceb in ?? ()
#88 0x00007fff59eecd0d in ?? ()
#89 0x00007fff59eecd1f in ?? ()
#90 0x00007fff59eecd71 in ?? ()
#91 0x00007fff59eecda9 in ?? ()
#92 0x00007fff59eecdf1 in ?? ()
#93 0x00007fff59eece52 in ?? ()
#94 0x00007fff59eece7f in ?? ()
#95 0x00007fff59eece8d in ?? ()
#96 0x00007fff59eecee2 in ?? ()
#97 0x00007fff59eecf2b in ?? ()
#98 0x00007fff59eecf42 in ?? ()
#99 0x00007fff59eecf53 in ?? ()
#100 0x00007fff59eecf68 in ?? ()
#101 0x00007fff59eecfb5 in ?? ()
#102 0x0000000000000000 in ?? ()

grigasp commented 1 year ago

The call stacks don't make much sense to me. Investigating this on my end...

markschlosseratbentley commented 1 year ago

In progress. Found other issues (hangs) in other tests as described above. Will try to run again in a loop before long.

markschlosseratbentley commented 1 year ago

Anyone know how long a test can "stall" before it is considered a hang and the CI system will terminate the run and consider it a failure? @grigasp @MichaelBelousov

markschlosseratbentley commented 1 year ago

Anyone know how long a test can "stall" before it is considered a hang and the CI system will terminate the run and consider it a failure? @grigasp @MichaelBelousov

@aruniverse do you happen to know?

MichaelBelousov commented 1 year ago

@markschlosseratbentley depends on the timeout set by the pipeline yaml

grigasp commented 1 year ago

@markschlosseratbentley depends on the timeout set by the pipeline yaml

Depends on what a "stall" means... generally, when a test hangs, it first hits a timeout set by mocha. However, there are kinds of hangs that mocha doesn't catch (e.g. ones that prevent the process from exiting rather than test completion), at which point the pipeline timeout comes into play.

markschlosseratbentley commented 1 year ago

@grigasp I have been reproducing the following driver error in rush cover on Ubuntu Linux locally, in the core-full-stack-tests section. It sits there for quite a while (not sure exactly how long), then eventually continues through the other tests. This makes me wonder if such an error is being hit by CI and is terminated for sitting there too long. Any thoughts?

--[ WARNING: core-full-stack-tests ]---------------[ 4 minutes 51.2 seconds ]--

WARNING: Tests attempted to load missing asset: "/locales/en-US/iModelJs.json"
WARNING: Tests attempted to load missing asset: "/locales/en-US/CoreTools.json"
WARNING: Tests attempted to load missing asset: "/locales/en/iModelJs.json"
WARNING: Tests attempted to load missing asset: "/locales/en/CoreTools.json"
WARNING: Tests attempted to load missing asset: "/images/imodeljs-icon.svg"
WARNING: Tests attempted to load missing asset: "/locales/en-US/Editor.json"
WARNING: Tests attempted to load missing asset: "/locales/en/Editor.json"
WARNING: Tests attempted to load missing asset: "/locales/en-US/TestApp.json"
WARNING: Tests attempted to load missing asset: "/locales/en/TestApp.json"
WARNING: Tests attempted to load missing asset: "/locales/en-US/MarkupTools.json"
WARNING: Tests attempted to load missing asset: "/locales/en/MarkupTools.json"
WARNING: Tests attempted to load missing asset: "/cursors/crosshair.cur"
WARNING: 2 tests skipped in core-full-stack-tests
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
(node:130306) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 itwin.ipcApp-notify listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit
(Use `exe --trace-warnings ...` to show where the warning was created)
WARNING: 2 tests skipped in core-full-stack-tests

--[ WARNING: presentation-full-stack-tests ]---------[ 1 minute 4.0 seconds ]--

WARNING: 1 test skipped in presentation-full-stack-tests

--[ WARNING: rpc-full-stack-tests ]-------------------------[ 25.14 seconds ]--

libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
WARNING: 1 test skipped in rpc-full-stack-tests
WARNING: 1 test skipped in rpc-full-stack-tests

rush cover (4 minutes 51.3 seconds)
Run 21

markschlosseratbentley commented 1 year ago

What I have read online is that the vaGetDriverNameByIndex libva error can be solved by updating the browser on Ubuntu machines. Do we know what browser version we are using for these tests in CI? How is that controlled? @wgoehrig any ideas?

grigasp commented 1 year ago

@grigasp I have been reproducing the following driver error in rush cover on Ubuntu Linux locally, in the core-full-stack-tests section. It sits there for quite a while (not sure exactly how long), then eventually continues through the other tests. This makes me wonder if such an error is being hit by CI and is terminated for sitting there too long. Any thoughts?

I believe logs of such a CI test run would give an answer to that.

markschlosseratbentley commented 1 year ago

@grigasp I have been reproducing the following driver error in rush cover on Ubuntu Linux locally, in the core-full-stack-tests section. It sits there for quite a while (not sure exactly how long), then eventually continues through the other tests. This makes me wonder if such an error is being hit by CI and is terminated for sitting there too long. Any thoughts?

I believe logs of such a CI test run would give an answer to that.

The logs for this particular issue have been purged it seems. Perhaps we should close this issue unless (or until) it recurs?

grigasp commented 1 year ago

I just checked iTwin.js pipeline failures since March 1 - there are two groups of failures:

markschlosseratbentley commented 1 year ago

@markschlosseratbentley make separate issues

iTwin / itwinjs-core