Open MichaelBelousov opened 2 years ago
I managed to get the exact unstripped addon that was released and got a symbolized (release-built) callstack. I can provide a core dump if necessary.
#0 0x00007ffd4a686638 in bcvContainerFree (pCont=0x0)
at /home/platform-bld/vsts_agent_work/242/s/iModelCore/BeSQLite/SQLite/blockcachevfs.c:3797
#1 0x00007ffd4a686ffb in bcvfsAttach (pFs=<optimized out>, zStorage=<optimized out>, zAccount=<optimized out>,
zContainer=<optimized out>, zAliasArg=<optimized out>, flags=<optimized out>, pzErr=<optimized out>)
at /home/platform-bld/vsts_agent_work/242/s/iModelCore/BeSQLite/SQLite/blockcachevfs.c:4027
#2 sqlite3_bcvfs_attach (pFs=0x559aca8, zStorage=<optimized out>, zAccount=<optimized out>, zCont=<optimized out>,
zAlias=<optimized out>, flags=2, pzErr=0x7fffffffc288)
at /home/platform-bld/vsts_agent_work/242/s/iModelCore/BeSQLite/SQLite/blockcachevfs.c:4130
#3 0x00007ffd4a663ec8 in BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3::operator()(char**) const (this=<optimized out>, msg=<optimized out>)
at /home/platform-bld/vsts_agent_work/242/s/iModelCore/BeSQLite/CloudSqlite.cpp:249
#4 std::_Function_handler<int (char**), BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3>::_M_invoke(std::_Any_data const&, char**&&) (__functor=..., __args=<optimized out>)
at /usr/lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1716
#5 0x00007ffd4a662923 in std::function<int (char**)>::operator()(char**) const (this=0x7fffffffc2e0,
__args=0x7fffffffc288) at /usr/lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:2127
#6 BentleyM0200::BeSQLite::CloudCache::CallSqliteFn(std::function<int (char**)>, char const*) (this=<optimized out>,
fn=..., funcName=0x7ffd4f10dabc "attach")
at /home/platform-bld/vsts_agent_work/242/s/iModelCore/BeSQLite/CloudSqlite.cpp:216
#7 0x00007ffd4a6631bd in BentleyM0200::BeSQLite::CloudContainer::Connect (this=<optimized out>, cache=...)
at /home/platform-bld/vsts_agent_work/242/s/iModelCore/BeSQLite/CloudSqlite.cpp:249
#8 0x00007ffd4a396fb3 in IModelJsNative::JsCloudContainer::Connect (this=0x5566210, info=...)
at /home/platform-bld/vsts_agent_work/242/s/iModelJsNodeAddon/JsCloudSqlite.cpp:197
#9 0x00007ffd4a397135 in Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect>(napi_env__*, napi_callback_info__*)::{lambda()#1}::operator()() const (
this=<optimized out>)
at /home/platform-bld/vsts_agent_work/242/b/LinuxX64/static/BuildContexts/iModelJsNodeAddon/VendorAPI/Napi/node-src/napi-inl.h:181
#10 Napi::details::WrapCallback<Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect>(napi_env__*, napi_callback_info__*)::{lambda()#1}>(Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect>(napi_env__*, napi_callback_info__*)::{lambda()#1}) (callback=...) at /home/platform-bld/vsts_agent_work/242/b/LinuxX64/static/BuildContexts/iModelJsNodeAddon/VendorAPI/Napi/node-src/napi-inl.h:79
#11 Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect> (env=<optimized out>, info=<optimized out>) at /home/platform-bld/vsts_agent_work/242/b/LinuxX64/static/BuildContexts/iModelJsNodeAddon/VendorAPI/Napi/node-src/napi-inl.h:178
#12 0x0000000000ab44dd in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) ()
#13 0x0000000000d53cfe in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
#14 0x0000000000d5511f in v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) ()
#15 0x00000000015f0b79 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
#16 0x000000000158354a in Builtins_InterpreterEntryTrampoline ()
#17 0x0000387807dc1599 in ?? ()
#18 0x00000c0207955f19 in ?? ()
#19 0x0000000600000000 in ?? ()
#20 0x0000387807dc1669 in ?? ()
#21 0x000014511e18f6a1 in ?? ()
#22 0x000014511e190871 in ?? ()
#23 0x000004574e956d99 in ?? ()
#24 0x000014511e18f1d1 in ?? ()
#25 0x000016ee5595d0c9 in ?? ()
#26 0x000014511e190871 in ?? ()
#27 0x00003d3d7c6482a1 in ?? ()
#28 0x00000c0207955f19 in ?? ()
#29 0x000014511e18f6a1 in ?? ()
#30 0x000000ea00000000 in ?? ()
#31 0x000016ee5595d299 in ?? ()
#32 0x0000000000000003 in ?? ()
#33 0x000010fbb67a0ce1 in ?? ()
#34 0x00000c020797c089 in ?? ()
#35 0x00007fffffffc748 in ?? ()
#36 0x0000000001580b85 in Builtins_JSConstructStubGeneric ()
Backtrace stopped: frame did not save the PC
I believe this is an indication of heap corruption somehow. I see nothing wrong in the failing code, and i can't see how bcvContainerFree
is ever called with 0x0. Is there any indication what's happening on the JavaScript side?
let me try stepping through js manually at addon load time to see if I can get any additional info on the js side
This was reproducible until instead of starting the CPU profiler I started it under the debugger. It now no longer reproduces, even after resetting the dependencies and repo state. Before that however I added some print debugging since the debugger wasn't working with the profiler, and I at least could see that it crashes after the addon is loaded and after the first test's describe
is called in core/backend/src/test/categories/Category.test.ts
, probably in the before
call but I couldn't find out before it stopped reproducing.
maybe it'll reproduce if I reinstall chrome :) I have no idea how to proceed. I suppose we can close this as having a workaround.
I just reproduced again by switching to transformer tests from core-backend tests. I'll try to get some more print debugging to figure out where
I confirmed via printf debugging that the connect
happens in javascript upon creating an empty snapshot, eventually cloudContainer.connect(this.workspace.cloudCache)
is called in the ITwinWorkspaceContainer
constructor upon adding a gcs workspace when loading the default databases for the "GeoCoordConfig" when making the new snapshot.
So that looks normal. I will see if the workaround works so I can get back to what I was doing but this may need further debugging on the native side for a real fix.
Make sure you have these changes:
https://github.com/iTwin/itwinjs-core/pull/4108 https://github.com/iTwin/itwinjs-core/pull/4095
I'm available if you want help diagnosing.
confirmed switching to clang-7 from clang-13 didn't help. I pulled native dependencies (apparently this also bumped me to db11a84ee9f812d6ca16f6abbf98408d01680f2c) I also had to remove core/transformer/lib/cjs/test/standalone/.cache.
Some combination of the above means I can now reproduce with a locally built release-mode addon, debug probably works too and now I should be able to figure out the state of the addon and why pCont = 0
new release stack:
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737348171712) at pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737348171712) at pthread_kill.c:80
#2 __GI___pthread_kill (threadid=140737348171712, signo=signo@entry=6) at pthread_kill.c:91
#3 0x00007ffff7a96476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff7a7c7b7 in __GI_abort () at abort.c:79
#5 0x00007ffff7a7c6db in __assert_fail_base (fmt=0x7ffff7c30770 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0x7ffd4e5dd4d9 "pCb->rc==SQLITE_OK && pCb->zErr==0",
file=0x7ffd4e5a2298 "/home/mike/work/itwinjs-cospace/repos/imodel02/imodel02/iModelCore/BeSQLite/SQLite/blockcachevfs.c", line=533, function=<optimized out>) at assert.c:92
#6 0x00007ffff7a8de26 in __GI___assert_fail (assertion=0x7ffd4e5dd4d9 "pCb->rc==SQLITE_OK && pCb->zErr==0",
file=0x7ffd4e5a2298 "/home/mike/work/itwinjs-cospace/repos/imodel02/imodel02/iModelCore/BeSQLite/SQLite/blockcachevfs.c", line=533, function=0x7ffd4e5dd4fc "void bcvfsFetchManifestCb(void *, int, char *, const u8 *, int)") at assert.c:101
#7 0x00007ffd48b38bdf in bcvfsFetchManifestCb ()
from /home/mike/work/imodeljs/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.4.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#8 0x00007ffd48b3da97 in bcvDispatchFree ()
from /home/mike/work/imodeljs/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.4.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#9 0x00007ffd489ba2d3 in bcvContainerFree ()
from /home/mike/work/imodeljs/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.4.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#10 0x00007ffd489bb211 in bcvfsAttach ()
from /home/mike/work/imodeljs/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.4.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#11 0x00007ffd489baed8 in sqlite3_bcvfs_attach ()
from /home/mike/work/imodeljs/common/temp/node_modules/.pnpm/@bentley+imodeljs-native@3.4.6/node_modules/@bentley/imodeljs-native/imodeljs-linux-x64/imodeljs.node
#12 0x00007ffd489716be in BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3::operator()(char**) const (this=0x7fffffffbec0, msg=0x7fffffffbdf0)
at /home/mike/work/itwinjs-cospace/repos/imodel02/imodel02/iModelCore/BeSQLite/CloudSqlite.cpp:249
#13 0x00007ffd489715f2 in std::__invoke_impl<int, BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3&, char**>(std::__invoke_other, BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3&, char**&&) (__f=..., __args=@0x7fffffffbd50: 0x7fffffffbdf0) at /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#14 0x00007ffd48971582 in std::__invoke_r<int, BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3&, char**>(BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3&, char**&&) (__fn=..., __args=@0x7fffffffbd50: 0x7fffffffbdf0) at /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114
#15 0x00007ffd48971462 in std::_Function_handler<int (char**), BentleyM0200::BeSQLite::CloudContainer::Connect(BentleyM0200::BeSQLite::CloudCache&)::$_3>::_M_invoke(std::_Any_data const&, char**&&) (__functor=..., __args=@0x7fffffffbd50: 0x7fffffffbdf0) at /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
#16 0x00007ffd4896d894 in std::function<int (char**)>::operator()(char**) const (this=0x7fffffffbec0, __args=0x7fffffffbdf0) at /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
#17 0x00007ffd4896c855 in BentleyM0200::BeSQLite::CloudCache::CallSqliteFn(std::function<int (char**)>, char const*) (this=0x50676c0, fn=..., funcName=0x7ffd4ea895cc "attach") at /home/mike/work/itwinjs-cospace/repos/imodel02/imodel02/iModelCore/BeSQLite/CloudSqlite.cpp:216
#18 0x00007ffd4896dc22 in BentleyM0200::BeSQLite::CloudContainer::Connect (this=0x55cbd10, cache=...) at /home/mike/work/itwinjs-cospace/repos/imodel02/imodel02/iModelCore/BeSQLite/CloudSqlite.cpp:249
#19 0x00007ffd482ab830 in IModelJsNative::JsCloudContainer::Connect (this=0x55cbcf0, info=...) at /home/mike/work/itwinjs-cospace/repos/imodel02/imodel02/iModelJsNodeAddon/JsCloudSqlite.cpp:197
#20 0x00007ffd482ac01f in Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect>(napi_env__*, napi_callback_info__*)::{lambda()#1}::operator()() const (this=0x7fffffffc158) at /home/mike/work/itwinjs-cospace/repos/imodel02/BuildOutput/LinuxX64/static/BuildContexts/iModelJsNodeAddon/VendorAPI/Napi/node-src/napi-inl.h:181
#21 0x00007ffd482abdf6 in Napi::details::WrapCallback<Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect>(napi_env__*, napi_callback_info__*)::{lambda()#1}>(Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect>(napi_env__*, napi_callback_info__*)::{lambda()#1}) (callback=...) at /home/mike/work/itwinjs-cospace/repos/imodel02/BuildOutput/LinuxX64/static/BuildContexts/iModelJsNodeAddon/VendorAPI/Napi/node-src/napi-inl.h:79
#22 0x00007ffd482aba5a in Napi::details::TemplatedInstanceVoidCallback<IModelJsNative::JsCloudContainer, &IModelJsNative::JsCloudContainer::Connect> (env=0x4bf73b0, info=0x7fffffffc1d0) at /home/mike/work/itwinjs-cospace/repos/imodel02/BuildOutput/LinuxX64/static/BuildContexts/iModelJsNodeAddon/VendorAPI/Napi/node-src/napi-inl.h:178
#23 0x0000000000ab44dd in v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) ()
#24 0x0000000000d53cfe in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
#25 0x0000000000d5511f in v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) ()
#26 0x00000000015f0b79 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
#27 0x000000000158354a in Builtins_InterpreterEntryTrampoline ()
#28 0x00000ae2c2f41599 in ?? ()
...
#47 0x0000000001580b85 in Builtins_JSConstructStubGeneric ()
Backtrace stopped: frame did not save the PC
looks like it now receives a SIGABRT from failing the assert in bcvfsFetchManifestCb
assert( pCb->rc==SQLITE_OK && pCb->zErr==0 );
~not sure why it was a segfault in the release build~ I guess the debug build aborts in the assert and that doesn't happen in the release build.
@kabentley still reproducing on the latest master itwinjs-core and native branch with those two PRs now definitely included on the js end:
node: /home/mike/work/itwinjs-cospace/repos/imodel02/imodel02/iModelCore/BeSQLite/SQLite/blockcachevfs.c:533: void bcvfsFetchManifestCb(void *, int, char *, const u8 *, int): Assertion `pCb->rc==SQLITE_OK && pCb->zErr==0' failed.
Aborted
I'm going to go find an addon that works for my CPU profiling and move on, but I think there are some people I can ask to take a look in the meanwhile.
@nick4598 can you see if any of the above makes sense to Dan?
Looks like gdb upon popping a frame occasionally handling a signal with v8::sample::SignalHandler::HandleProfilerSignal
from the chrome CPU profiler is the SIGPROF signal which I was able to confirm from the siginfo_t
passed to the handler. That makes sense, I'm just not familiar with profiler implementation. This signal is sent at some frequency (maybe by the OS, didn't look into how) for the profiler to use to gather (js) stacks and do its job.
I'm not sure if this interacts with the async (c-ares) curl dns resolution implementation, but I could not reproduce locally until I consumed the version of the curl native dependency that uses c-ares instead of the default non-threadsafe signal-based curl implementation of dns resolution.
running into this while trying to set up profiling of an iTwin.js process. Might have to try using windows as a workaround or look into this.
I have a fix for bcvfs's abort which was a dangling stack reference, I gave details to @nick4598 to pass on to Dan. But I haven't been able to pin down why curl doesn't successfully request. Verbose curl logging doesn't say anything and c-ares doesn't seem to have its own logging. I do see timeouts but I'm not sure if that's because of me stepping through code too slowly. Both gdb
and lldb
crash on parts of this code.
Describe the bug Segfault in
CloudContainer::Connect
(see additional context for stack) when taking javascript CPU profile of transformer in chrome devtools. This occurs on Ubuntu Linux, I have not yet tried any other OSes.This means to measure performance differences in changes I'm making I need to go back further on master to find a working addon which I will do soon.
To Reproduce
Expected behavior No segfault, able to use chrome devtools to produce javascript CPU profiles of iTwin.js code.
Screenshots N/A
Desktop (please complete the applicable information):
Additional context
I could not reproduce with a locally built debug and non-debug addon. Any pointers for what might be different between the released addon and my local build of b0e8520af50a86ef02c6dd660259dda9d84ab14d with DEBUG and then NDEBUG? Also if someone could confirm that b0e8520af50a86ef02c6dd660259dda9d84ab14d is the corresponding commit to
@bentley/imodeljs-native@3.4.6
I would appreciate that.Stack from attaching via gdb to the process:
gdb also gives me these hundreds of these warnings if someone knows what it means. I haven't looked into it yet.