Expensify / Bedrock

Rock solid distributed database specializing in active/active automatic failover and WAN replication
https://bedrockdb.com
GNU Lesser General Public License v3.0
1.1k stars 82 forks source link

Only log the first and last 20 frames of large stack traces. #1737

Closed tylerkaraszewski closed 4 months ago

tylerkaraszewski commented 4 months ago

Details

Note, there are (annoyingly) two places we log stack traces. One is _SSignal_StackTrace (handled in this PR) and the other is SLogStackTrace (not handled in this PR).

I think we could remove SLogStackTrace but I don't think we need to do it now.

Fixed Issues

Fixes slack canvas: https://expensify.enterprise.slack.com/canvas/C0714QF3A1Z $ https://github.com/Expensify/Expensify/issues/397250

Tests

To test this I added the following code to main.cpp:

int addForever2(int from);
int addForever1(int from) {
    if (from == INT_MAX) {
        return from;
    }
    return addForever2(from + 1);
}
int addForever2(int from) {
    if (from == INT_MAX) {
        return from;
    }
    return addForever1(from + 1);
}

There are two mutually recursive functions simply because my dev enviroment logs a single line and then <repeated 79 times> for multiple identical loglines in a row. this forces each line to be different from the previous.

Calling this from main generates the following logs:

2024-05-17T17:47:35.292141+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:185) _SSignal_StackTrace [main] [warn] Signal Segmentation fault(11) caused crash, logging stack trace.
2024-05-17T17:47:35.407217+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:209) _SSignal_StackTrace [main] [warn] Stack depth is 261955 only logging first and last 20 frames.
2024-05-17T17:47:35.409294+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #0: _SSignal_StackTrace(int, siginfo_t*, void*)
2024-05-17T17:47:35.409391+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #1: __kernel_rt_sigreturn
2024-05-17T17:47:35.409454+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #2: addForever2(int)
2024-05-17T17:47:35.409602+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #3: addForever1(int)
2024-05-17T17:47:35.409670+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #4: addForever2(int)
2024-05-17T17:47:35.409752+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #5: addForever1(int)
2024-05-17T17:47:35.409821+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #6: addForever2(int)
2024-05-17T17:47:35.409950+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #7: addForever1(int)
2024-05-17T17:47:35.410020+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #8: addForever2(int)
2024-05-17T17:47:35.410058+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #9: addForever1(int)
2024-05-17T17:47:35.410140+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #10: addForever2(int)
2024-05-17T17:47:35.410248+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #11: addForever1(int)
2024-05-17T17:47:35.410323+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #12: addForever2(int)
2024-05-17T17:47:35.410396+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #13: addForever1(int)
2024-05-17T17:47:35.410494+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #14: addForever2(int)
2024-05-17T17:47:35.410640+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #15: addForever1(int)
2024-05-17T17:47:35.410721+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #16: addForever2(int)
2024-05-17T17:47:35.410801+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #17: addForever1(int)
2024-05-17T17:47:35.410852+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #18: addForever2(int)
2024-05-17T17:47:35.410934+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #19: addForever1(int)
2024-05-17T17:47:35.411261+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261935: addForever1(int)
2024-05-17T17:47:35.411347+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261936: addForever2(int)
2024-05-17T17:47:35.411411+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261937: addForever1(int)
2024-05-17T17:47:35.411455+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261938: addForever2(int)
2024-05-17T17:47:35.411490+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261939: addForever1(int)
2024-05-17T17:47:35.411569+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261940: addForever2(int)
2024-05-17T17:47:35.411733+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261941: addForever1(int)
2024-05-17T17:47:35.411804+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261942: addForever2(int)
2024-05-17T17:47:35.411868+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261943: addForever1(int)
2024-05-17T17:47:35.411926+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261944: addForever2(int)
2024-05-17T17:47:35.411989+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261945: addForever1(int)
2024-05-17T17:47:35.412102+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261946: addForever2(int)
2024-05-17T17:47:35.412180+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261947: addForever1(int)
2024-05-17T17:47:35.412250+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261948: addForever2(int)
2024-05-17T17:47:35.412360+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261949: addForever1(int)
2024-05-17T17:47:35.412465+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261950: addForever2(int)
2024-05-17T17:47:35.412596+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261951: addForever1(int)
2024-05-17T17:47:35.412681+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261952: main
2024-05-17T17:47:35.412801+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261953: __libc_start_main
2024-05-17T17:47:35.412879+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:229) _SSignal_StackTrace [main] [warn] Frame #261954: ./bedrock(+0x133b18) [0xaaaad161bb18]
2024-05-17T17:47:35.412988+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:237) _SSignal_StackTrace [main] [warn] Calling DIE function.
2024-05-17T17:47:35.413056+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:240) _SSignal_StackTrace [main] [warn] DIE function returned.
2024-05-17T17:47:35.413085+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:249) _SSignal_StackTrace [main] [warn] Aborting.
2024-05-17T17:47:35.413115+00:00 expensidev2004 bedrock: xxxxxx (SSignal.cpp:252) _SSignal_StackTrace [main] [warn] Already in ABORT.

Undoing the change in main and running the BadCommand test still generates reasonable stacks. Example 1 from a segfault:

2024-05-17T17:49:43.861791+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:185) _SSignal_StackTrace [socket6] [warn] Signal Segmentation fault(11) caused crash, logging stack trace.
2024-05-17T17:49:43.862339+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #0: _SSignal_StackTrace(int, siginfo_t*, void*)
2024-05-17T17:49:43.862393+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #1: __kernel_rt_sigreturn
2024-05-17T17:49:43.862471+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #2: TestPluginCommand::peek(SQLite&)
2024-05-17T17:49:43.862513+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #3: BedrockCore::peekCommand(std::unique_ptr<BedrockCommand, std::default_delete<BedrockCommand> >&, bool)
2024-05-17T17:49:43.862676+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #4: BedrockServer::runCommand(std::unique_ptr<BedrockCommand, std::default_delete<BedrockCommand> >&&, bool, bool)
2024-05-17T17:49:43.862786+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #5: BedrockServer::handleSocket(STCPManager::Socket&&, bool, bool, bool)
2024-05-17T17:49:43.862990+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #6: void std::__invoke_impl<void, void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>(std::__invoke_memfun_deref, void (BedrockServer::*&&)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*&&, STCPManager::Socket&&, bool&&, bool&&, bool&&)
2024-05-17T17:49:43.863192+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #7: std::__invoke_result<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>::type std::__invoke<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>(void (BedrockServer::*&&)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*&&, STCPManager::Socket&&, bool&&, bool&&, bool&&)
2024-05-17T17:49:43.863372+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #8: void std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>)
2024-05-17T17:49:43.863510+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #9: std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> >::operator()()
2024-05-17T17:49:43.863879+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #10: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> > >::_M_run()
2024-05-17T17:49:43.863950+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #11: /lib/aarch64-linux-gnu/libstdc++.so.6(+0xdbc6c) [0xffff86f7bc6c]
2024-05-17T17:49:43.864048+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #12: /lib/aarch64-linux-gnu/libpthread.so.0(+0x7624) [0xffff87138624]
2024-05-17T17:49:43.864220+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:229) _SSignal_StackTrace [socket6] [warn] Frame #13: /lib/aarch64-linux-gnu/libc.so.6(+0xd162c) [0xffff86d2862c]
2024-05-17T17:49:43.864329+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:237) _SSignal_StackTrace [socket6] [warn] Calling DIE function.
2024-05-17T17:49:43.866629+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:240) _SSignal_StackTrace [socket6] [warn] DIE function returned.
2024-05-17T17:49:43.866830+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:242) _SSignal_StackTrace [socket6] [warn] Killing peer connections.
2024-05-17T17:49:43.867177+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:249) _SSignal_StackTrace [socket6] [warn] Aborting.
2024-05-17T17:49:43.867225+00:00 expensidev2004 bedrock10001: ibfdUf (SSignal.cpp:252) _SSignal_StackTrace [socket6] [warn] Already in ABORT.

Example 2 from an assertion:

2024-05-17T17:49:54.031277+00:00 expensidev2004 bedrock10001: F9oHaz (TestPlugin.cpp:282) peek [socket3] [eror] Assertion failed: (0) != true
2024-05-17T17:49:54.033330+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn]
2024-05-17T17:49:54.033598+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] SLogStackTrace(int) [0xaaaadc66b798]
2024-05-17T17:49:54.033937+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] TestPluginCommand::peek(SQLite&) [0xffff89c32f9c]
2024-05-17T17:49:54.034025+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] BedrockCore::peekCommand(std::unique_ptr<BedrockCommand, std::default_delete<BedrockCommand> >&, bool) [0xaaaadc649638]
2024-05-17T17:49:54.034097+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] BedrockServer::runCommand(std::unique_ptr<BedrockCommand, std::default_delete<BedrockCommand> >&&, bool, bool) [0xaaaadc503f54]
2024-05-17T17:49:54.034149+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] BedrockServer::handleSocket(STCPManager::Socket&&, bool, bool, bool) [0xaaaadc5202b8]
2024-05-17T17:49:54.034197+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] void std::__invoke_impl<void, void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>(std::__invoke_memfun_deref, void (BedrockServer::*&&)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*&&, STCPManager::Socket&&, bool&&, bool&&, bool&&) [0xaaaadc54bd4c]
2024-05-17T17:49:54.034238+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] std::__invoke_result<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>::type std::__invoke<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>(void (BedrockServer::*&&)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*&&, STCPManager::Socket&&, bool&&, bool&&, bool&&) [0xaaaadc54b938]
2024-05-17T17:49:54.034322+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] void std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>) [0xaaaadc54b398]
2024-05-17T17:49:54.034442+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> >::operator()() [0xaaaadc54af84]
2024-05-17T17:49:54.034486+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> > >::_M_run() [0xaaaadc54a484]
2024-05-17T17:49:54.034538+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] /lib/aarch64-linux-gnu/libstdc++.so.6(+0xdbc6c) [0xffff8a80ac6c]
2024-05-17T17:49:54.034579+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] /lib/aarch64-linux-gnu/libpthread.so.0(+0x7624) [0xffff8a9c7624]
2024-05-17T17:49:54.034676+00:00 expensidev2004 bedrock10001: F9oHaz (SLog.cpp:24) SLogStackTrace [socket3] [warn] /lib/aarch64-linux-gnu/libc.so.6(+0xd162c) [0xffff8a5b762c]
2024-05-17T17:49:54.034757+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:185) _SSignal_StackTrace [socket3] [warn] Signal Aborted(6) caused crash, logging stack trace.
2024-05-17T17:49:54.034799+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #0: _SSignal_StackTrace(int, siginfo_t*, void*)
2024-05-17T17:49:54.034839+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #1: __kernel_rt_sigreturn
2024-05-17T17:49:54.034880+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #2: gsignal
2024-05-17T17:49:54.034923+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #3: abort
2024-05-17T17:49:54.034963+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #4: TestPluginCommand::peek(SQLite&)
2024-05-17T17:49:54.035048+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #5: BedrockCore::peekCommand(std::unique_ptr<BedrockCommand, std::default_delete<BedrockCommand> >&, bool)
2024-05-17T17:49:54.035096+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #6: BedrockServer::runCommand(std::unique_ptr<BedrockCommand, std::default_delete<BedrockCommand> >&&, bool, bool)
2024-05-17T17:49:54.035143+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #7: BedrockServer::handleSocket(STCPManager::Socket&&, bool, bool, bool)
2024-05-17T17:49:54.035186+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #8: void std::__invoke_impl<void, void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>(std::__invoke_memfun_deref, void (BedrockServer::*&&)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*&&, STCPManager::Socket&&, bool&&, bool&&, bool&&)
2024-05-17T17:49:54.035270+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #9: std::__invoke_result<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>::type std::__invoke<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool>(void (BedrockServer::*&&)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*&&, STCPManager::Socket&&, bool&&, bool&&, bool&&)
2024-05-17T17:49:54.035325+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #10: void std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>)
2024-05-17T17:49:54.035364+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #11: std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> >::operator()()
2024-05-17T17:49:54.035437+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #12: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (BedrockServer::*)(STCPManager::Socket&&, bool, bool, bool), BedrockServer*, STCPManager::Socket, bool, bool, bool> > >::_M_run()
2024-05-17T17:49:54.035487+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #13: /lib/aarch64-linux-gnu/libstdc++.so.6(+0xdbc6c) [0xffff8a80ac6c]
2024-05-17T17:49:54.035546+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #14: /lib/aarch64-linux-gnu/libpthread.so.0(+0x7624) [0xffff8a9c7624]
2024-05-17T17:49:54.035615+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:229) _SSignal_StackTrace [socket3] [warn] Frame #15: /lib/aarch64-linux-gnu/libc.so.6(+0xd162c) [0xffff8a5b762c]
2024-05-17T17:49:54.035729+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:237) _SSignal_StackTrace [socket3] [warn] Calling DIE function.
2024-05-17T17:49:54.039771+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:240) _SSignal_StackTrace [socket3] [warn] DIE function returned.
2024-05-17T17:49:54.039861+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:242) _SSignal_StackTrace [socket3] [warn] Killing peer connections.
2024-05-17T17:49:54.040025+00:00 expensidev2004 bedrock10001: F9oHaz (SSignal.cpp:252) _SSignal_StackTrace [socket3] [warn] Already in ABORT.

Internal Testing Reminder: when changing bedrock, please compile auth against your new changes