dlex
commented
2 years ago ASAN has detected heap-use-after-free during RP shutdown, when many services including raft::group_manager have been stopped and destroyed, but reactor still runs. Adopted stack trace:
std::exception_ptr::operator bool()
ss::basic_semaphore<>::signal(unsigned long) at 'if (_ex)'
ss::semaphore_units<>::~semaphore_units() at '_sem->signal(_n);'
std::vector<ss::semaphore_units<>>::~vector()
rpc::transport::entry::~entry()
rpc::transport::fail_outstanding_futures() at '_requests_queue.clear();'
net::base_transport::stop()
rpc::reconnect_transport::stop()::$_1::operator()() constData structure containment:
void* exception_ptr::_M_exception_object; <-- the invalid pointer
std::exception_ptr basic_semaphore::_ex;
ss:semaphore<>* ss::semaphore_units::_sem; <-- may be broken in first place
ss::foreign_ptr<ss::lw_shared_ptr<std::vector<ss::semaphore_units<>>>> rpc::transport::entry::resource_units;
absl::btree_map<sequence_t, std::unique_ptr<transport::entry>> rpc::transport::_requests_queue;ss::foreign_ptr is destroyed synchronously because the dtor is called in the same shard where the ptr was constructed.
Allocation and deallocation stacks of the accessed memory block point to raft::group_manager which doesn't make sense - it it not the object that would store the exception_object the semaphore points to. Most probably it is the semaphore object itself who had been destroyed before it was used in this case. Since rpc::transport::entry stores semaphore units supplied by an RPC caller, it is not practical to try to determine what was the missing semaphore object in this case.
A possible strategy to resolve this case is to let it go and hope that the next time ASAN will trigger the error while accessing the semaphore object memory (the basic_semaphore::_ex member), and then the alloc/free stack traces will point to the missing semaphore.
Another option is to increase the probability for ASAN to triggier the error at the right place is to add the code to ss::semaphore_units::~semaphore_units() that will explicitly read sizeof(ss::semaphore<>) bytes pointed by `ss::semaphore_units::_sem' in debug build.
ztlpn
mmaslankaprv
bharathv
https://buildkite.com/redpanda/redpanda/builds/11804#0181ab92-27a7-4657-864e-2ac060cde87a/1575-8093