mversche
commented
2 years ago Internal tracking 170729991
Closed greghogan closed 1 year ago
mversche
commented
2 years ago Internal tracking 170729991
lalawawa
commented
2 years ago Greg Hogan, it would help a lot if you would mention what operating system and architecture this occurred on.
lalawawa
commented
2 years ago OK, I've been told this was on Amazon Linux.
greghogan
commented
2 years ago If extra details are helpful, hardware is an AWS EC2 r6i.xlarge instance. As noted above, the OS is Amazon Linux 2 but the build system is GNU Guix.
lalawawa
commented
2 years ago x86 stacks are hard to walk back when you want to do a stack trace because, potentially, there are near and far calls and it's awkward to distinguish them. So Bloomberg doesn't provide it's own stack walkback function on x86, we use functions provided by the OS, Windows or Linux. The function we use on Linux is 'backtrace'. One problem with stack walkback functions is that sometimes they are "narcissic", that is, they report stack frames including the stack walkback function itself. The ones we write at BB for non-x86 architectures don't do that, but fox x86 we have to adjust for narcissicm by ignoring the frame at the top sometimes. It looks like in Amazon Linux someone re-wrote 'backtrace' to report one more narcissic frame than we were expecting, so we're not ignoring enough frames so there's this garbage frame on top of the stack trace throwing off the tests. I can't test on your architecture, but one thing you might want to try is to change 'k_IGNORE_FRAMES' in bde/groups/bsl/bsls/bsls_stackaddressutil.h line 317 from '1' to '2' and see if that helps.
lalawawa
commented
2 years ago Actually, now that I'm looking at it more, I'm not sure that's what's going on. Because rather than a garbage from on the stack top, a frame with 'getStackAddresses' should be visible above the garbage frame.
lalawawa
commented
2 years ago Have you somehow built bsl and bal with different build flags, so that libbsl.a has debugging symbols stripped?
lalawawa
commented
2 years ago It looks like all symbols in bsl are missing. When we build at Bloomberg, we never build on any form of Unix without debug symbols.
mversche
commented
2 years ago @greghogan You mentioned you are using the "GNU Guix" build system. I'm not familiar with it, but my read is that it is a package manager? i.e., you are actually performing the build using the tools in bde-tools, which uses CMake/ninja to compile with gcc (or possibly clang)
Could you provide the series of steps you used to perform the build and their output (verbose if possible) ?
greghogan
commented
1 year ago The following build log shows the build commands; search for the lines beginning with starting phase (the interesting phase start with 'configure'). This is using CMake and GCC but not the Python script from bde-tools.
zhl6k25qlxsb60m5z4342n1jn17pw7-bloomberg-bde-3.110.0.3.drv.gz
The UFID is set to opt_dbg_exc_mt_64_shr_cpp17 and libbsl.so is unstripped.
$ file libbsl.so
libbsl.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, with debug_info, not stripped$ ./balst_stacktraceutil.t 3 -v -v -v
TEST /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp CASE 3
TEST OF DIRECT CALL TO getStackAddresses
========================================
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(416): (PLAT_WIN && !DEBUG_ON) || pastMain || frame.isSymbolNameKnown() (failed)
i: 0
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(420): pastMain || frame.isOffsetFromSymbolKnown() (failed)
i: 0 offset: 18446744073709551615
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(425): reachedMain || offset < maxOffset (failed)
str.length() = 494
str = (0): --unknown-- at 0x7f840278e38f
(1): _ZL10case_3_Topb+0x78 at 0x40eb28 source:balst_stacktraceutil.t.cpp in balst_stacktraceutil.t
(2): _ZN6CASE_311upperMiddleEb+0x36 at 0x40bf06 in balst_stacktraceutil.t
(3): _ZN6CASE_311lowerMiddleEb+0x32 at 0x40bf72 in balst_stacktraceutil.t
(4): _ZN6CASE_36bottomEb+0x4a at 0x40bffa in balst_stacktraceutil.t
(5): main+0xe5b at 0x4097cb in balst_stacktraceutil.t
(6): --unknown-- at 0x7f84021e97dd
(7): _start+0x2a at 0x40b87a in balst_stacktraceutil.t
(0): --unknown-- at 0x7f840278e38f
(1): _ZL10case_3_Topb+0x78 at 0x40eb28 source:balst_stacktraceutil.t.cpp in balst_stacktraceutil.t
(2): _ZN6CASE_311upperMiddleEb+0x36 at 0x40bf06 in balst_stacktraceutil.t
(3): _ZN6CASE_311lowerMiddleEb+0x32 at 0x40bf72 in balst_stacktraceutil.t
(4): _ZN6CASE_36bottomEb+0x4a at 0x40bffa in balst_stacktraceutil.t
(5): main+0xe5b at 0x4097cb in balst_stacktraceutil.t
(6): --unknown-- at 0x7f84021e97dd
(7): _start+0x2a at 0x40b87a in balst_stacktraceutil.t
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(416): (PLAT_WIN && !DEBUG_ON) || pastMain || frame.isSymbolNameKnown() (failed)
i: 0
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(420): pastMain || frame.isOffsetFromSymbolKnown() (failed)
i: 0 offset: 18446744073709551615
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(425): reachedMain || offset < maxOffset (failed)
str.length() = 495
str = (0): --unknown-- at 0x7f840278e38f
(1): case_3_Top(bool)+0x78 at 0x40eb28 source:balst_stacktraceutil.t.cpp in balst_stacktraceutil.t
(2): CASE_3::upperMiddle(bool)+0x36 at 0x40bf06 in balst_stacktraceutil.t
(3): CASE_3::lowerMiddle(bool)+0x32 at 0x40bf72 in balst_stacktraceutil.t
(4): CASE_3::bottom(bool)+0x4a at 0x40bffa in balst_stacktraceutil.t
(5): main+0xe7b at 0x4097eb in balst_stacktraceutil.t
(6): --unknown-- at 0x7f84021e97dd
(7): _start+0x2a at 0x40b87a in balst_stacktraceutil.t
(0): --unknown-- at 0x7f840278e38f
(1): case_3_Top(bool)+0x78 at 0x40eb28 source:balst_stacktraceutil.t.cpp in balst_stacktraceutil.t
(2): CASE_3::upperMiddle(bool)+0x36 at 0x40bf06 in balst_stacktraceutil.t
(3): CASE_3::lowerMiddle(bool)+0x32 at 0x40bf72 in balst_stacktraceutil.t
(4): CASE_3::bottom(bool)+0x4a at 0x40bffa in balst_stacktraceutil.t
(5): main+0xe7b at 0x4097eb in balst_stacktraceutil.t
(6): --unknown-- at 0x7f84021e97dd
(7): _start+0x2a at 0x40b87a in balst_stacktraceutil.t
Error, non-zero test status = 6.$ ./balst_stacktraceutil.t 14 -v -v -v
TEST /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp CASE 14
TESTING: Stack Trace With Many Components
=========================================
mangledSearch = getStackAddresses, sizeof(ptr) = 8, sizeof(funcPtr) = 8, ptr = 0x7fce55df0381
mangledSearch = logFormattedMessage, sizeof(ptr) = 8, sizeof(funcPtr) = 8, ptr = 0x7fce55def091
mangledSearch = address, sizeof(ptr) = 8, sizeof(funcPtr) = 16, ptr = 0x413c91
mangledSearch = strstrCaseless, sizeof(ptr) = 8, sizeof(funcPtr) = 8, ptr = 0x7fce563603f1
mangledSearch = upperCaseCmp, sizeof(ptr) = 8, sizeof(funcPtr) = 8, ptr = 0x7fce56360aa1
mangledSearch = allocateAndExpand, sizeof(ptr) = 8, sizeof(funcPtr) = 16, ptr = 0x413db1
mangledSearch = unmap, sizeof(ptr) = 8, sizeof(funcPtr) = 8, ptr = 0x7fce563c32f1
mangledSearch = compareText, sizeof(ptr) = 8, sizeof(funcPtr) = 8, ptr = 0x7fce55de3311
mangledSearch = numBlocksInUse, sizeof(ptr) = 8, sizeof(funcPtr) = 16, ptr = 0x413d71
mangledSearch = exit, sizeof(ptr) = 8, sizeof(funcPtr) = 8, ptr = 0x413da1
mangledSearch = accumulatedTimes, sizeof(ptr) = 8, sizeof(funcPtr) = 16, ptr = 0x7fce55df0451
mangledSearch = totalNanoseconds, sizeof(ptr) = 8, sizeof(funcPtr) = 16, ptr = 0x413c71
LINE: 2171 expName: BloombergLP::bsls::StackAddressUtil::getStackAddresses(void
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2263): (passed.d_symbolPresent = !frame.symbolName().empty()) (failed)
LINE: 2171 expName: BloombergLP::bsls::StackAddressUtil::getStackAddresses(void name:
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2274): (passed.d_symbolGood = 0 == bsl::strncmp(expName.c_str(), name, expName.length())) (failed)
LINE: 2171 mangledName: expMangled: getStackAddresses
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2280): (passed.d_mangledSymbolGood = npos != frame.mangledSymbolName().find("BloombergLP")) (failed)
LINE: 2171 mangledName: expMangled: getStackAddresses
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2283): (passed.d_mangledSymbolGood &= npos != frame.mangledSymbolName().find(expMangled)) (failed)
Line number / source file name test skipped on this platform.
BloombergLP::bsls::StackAddressUtil::getStackAddresses(void failed
LINE: 2174 expName: BloombergLP::bsls::Log::logFormattedMessage(
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2263): (passed.d_symbolPresent = !frame.symbolName().empty()) (failed)
LINE: 2174 expName: BloombergLP::bsls::Log::logFormattedMessage( name:
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2274): (passed.d_symbolGood = 0 == bsl::strncmp(expName.c_str(), name, expName.length())) (failed)
LINE: 2174 mangledName: expMangled: logFormattedMessage
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2280): (passed.d_mangledSymbolGood = npos != frame.mangledSymbolName().find("BloombergLP")) (failed)
LINE: 2174 mangledName: expMangled: logFormattedMessage
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2283): (passed.d_mangledSymbolGood &= npos != frame.mangledSymbolName().find(expMangled)) (failed)
BloombergLP::bsls::Log::logFormattedMessage( failed
BloombergLP::balst::StackTraceFrame::address() passed
LINE: 2180 expName: BloombergLP::bdlb::String::strstrCaseless(
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2263): (passed.d_symbolPresent = !frame.symbolName().empty()) (failed)
LINE: 2180 expName: BloombergLP::bdlb::String::strstrCaseless( name:
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2274): (passed.d_symbolGood = 0 == bsl::strncmp(expName.c_str(), name, expName.length())) (failed)
LINE: 2180 mangledName: expMangled: strstrCaseless
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2280): (passed.d_mangledSymbolGood = npos != frame.mangledSymbolName().find("BloombergLP")) (failed)
LINE: 2180 mangledName: expMangled: strstrCaseless
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2283): (passed.d_mangledSymbolGood &= npos != frame.mangledSymbolName().find(expMangled)) (failed)
BloombergLP::bdlb::String::strstrCaseless( failed
LINE: 2183 expName: BloombergLP::bdlb::StringRefUtil::upperCaseCmp(
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2263): (passed.d_symbolPresent = !frame.symbolName().empty()) (failed)
LINE: 2183 expName: BloombergLP::bdlb::StringRefUtil::upperCaseCmp( name:
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2274): (passed.d_symbolGood = 0 == bsl::strncmp(expName.c_str(), name, expName.length())) (failed)
LINE: 2183 mangledName: expMangled: upperCaseCmp
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2280): (passed.d_mangledSymbolGood = npos != frame.mangledSymbolName().find("BloombergLP")) (failed)
LINE: 2183 mangledName: expMangled: upperCaseCmp
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2283): (passed.d_mangledSymbolGood &= npos != frame.mangledSymbolName().find(expMangled)) (failed)
BloombergLP::bdlb::StringRefUtil::upperCaseCmp( failed
BloombergLP::bdlma::SequentialAllocator::allocateAndExpand( passed
LINE: 2189 expName: BloombergLP::bdls::FilesystemUtil::unmap(void
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2263): (passed.d_symbolPresent = !frame.symbolName().empty()) (failed)
LINE: 2189 expName: BloombergLP::bdls::FilesystemUtil::unmap(void name:
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2274): (passed.d_symbolGood = 0 == bsl::strncmp(expName.c_str(), name, expName.length())) (failed)
LINE: 2189 mangledName: expMangled: unmap
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2280): (passed.d_mangledSymbolGood = npos != frame.mangledSymbolName().find("BloombergLP")) (failed)
LINE: 2189 mangledName: expMangled: unmap
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2283): (passed.d_mangledSymbolGood &= npos != frame.mangledSymbolName().find(expMangled)) (failed)
BloombergLP::bdls::FilesystemUtil::unmap(void failed
LINE: 2192 expName: BloombergLP::bslim::TestUtil::compareText(BloombergLP::bslstl::
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2263): (passed.d_symbolPresent = !frame.symbolName().empty()) (failed)
LINE: 2192 expName: BloombergLP::bslim::TestUtil::compareText(BloombergLP::bslstl:: name:
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2274): (passed.d_symbolGood = 0 == bsl::strncmp(expName.c_str(), name, expName.length())) (failed)
LINE: 2192 mangledName: expMangled: compareText
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2280): (passed.d_mangledSymbolGood = npos != frame.mangledSymbolName().find("BloombergLP")) (failed)
LINE: 2192 mangledName: expMangled: compareText
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2283): (passed.d_mangledSymbolGood &= npos != frame.mangledSymbolName().find(expMangled)) (failed)
BloombergLP::bslim::TestUtil::compareText(BloombergLP::bslstl:: failed
BloombergLP::bslma::TestAllocator::numBlocksInUse() passed
BloombergLP::bslmt::ThreadUtil::exit(void passed
LINE: 2201 expName: BloombergLP::bsls::Stopwatch::accumulatedTimes(double
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2263): (passed.d_symbolPresent = !frame.symbolName().empty()) (failed)
LINE: 2201 expName: BloombergLP::bsls::Stopwatch::accumulatedTimes(double name:
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2274): (passed.d_symbolGood = 0 == bsl::strncmp(expName.c_str(), name, expName.length())) (failed)
LINE: 2201 mangledName: expMangled: accumulatedTimes
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2280): (passed.d_mangledSymbolGood = npos != frame.mangledSymbolName().find("BloombergLP")) (failed)
LINE: 2201 mangledName: expMangled: accumulatedTimes
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bal/balst/balst_stacktraceutil.t.cpp(2283): (passed.d_mangledSymbolGood &= npos != frame.mangledSymbolName().find(expMangled)) (failed)
BloombergLP::bsls::Stopwatch::accumulatedTimes(double failed
BloombergLP::bsls::TimeInterval::totalNanoseconds() passed
Totals: 5 passed 7 failed.
( 0): sp(0) msg(0) sg(0) sfn(0) fp(0) l(0) :-1
( 1): sp(0) msg(0) sg(0) sfn(0) fp(0) l(0) :-1
( 2): sp(1) msg(1) sg(1) sfn(0) fp(0) l(0) BloombergLP::balst::StackTraceFrame::address() const :-1
( 3): sp(0) msg(0) sg(0) sfn(0) fp(0) l(0) :-1
( 4): sp(0) msg(0) sg(0) sfn(0) fp(0) l(0) :-1
( 5): sp(1) msg(1) sg(1) sfn(0) fp(0) l(0) BloombergLP::bdlma::SequentialAllocator::allocateAndExpand(unsigned long*) :-1
( 6): sp(0) msg(0) sg(0) sfn(0) fp(0) l(0) :-1
( 7): sp(0) msg(0) sg(0) sfn(0) fp(0) l(0) :-1
( 8): sp(1) msg(1) sg(1) sfn(0) fp(0) l(0) BloombergLP::bslma::TestAllocator::numBlocksInUse() const :-1
( 9): sp(1) msg(1) sg(1) sfn(0) fp(0) l(0) BloombergLP::bslmt::ThreadUtil::exit(void*) :-1
(10): sp(0) msg(0) sg(0) sfn(0) fp(0) l(0) :-1
(11): sp(1) msg(1) sg(1) sfn(0) fp(0) l(0) BloombergLP::bsls::TimeInterval::totalNanoseconds() const :-1
Error, non-zero test status = 28.Did you follow my advice and increment 'k_IGNORE_FRAMES' in bsls_stackaddressutil.h by 1?
lalawawa
commented
1 year ago Regarding case 14, that one is just taking a lot of function ptrs and resolving them, rather the getting return addresses off the stack. On some platforms, if you take a function ptr, you don't get a ptr to the actual function, you get a ptr to a little thunk that will call or jump to the function. Maybe that's what's going on?
lalawawa
commented
1 year ago Read the comment at the start of case 14 about why we disable it on some versions of Windows. I suggest you change the #if statement to disable that test on your platform as well. It's not a real important test. Looking at the output you're posting, while you're getting a garbage frame at the top of the stack (which you can get rid of by increasing 'k_IGNORE_FRAMES', you're getting pretty usable stack traces.
balst_stacktraceutil.t.cpp is a super-thorough test of the stack trace facility, and it's thoroughly riddled with #ifdef's to accommodate all the quirks of different platforms. Since I can't build on your platform, I can't correct the #ifdef's to accommodate your platform.
bde 3.110.0.3 bde-tools 3.110.0.0