search

Netflix-Skunkworks / spectatord

A high performance metrics daemon
Apache License 2.0
24 stars 5 forks source link

Remove usage of tcmalloc #25

Closed dmuino closed 3 years ago

dmuino commented 3 years ago

Trying to narrow down the issue of a stuck spectatord process

dmuino commented 3 years ago

For future reference, an instance of the process in a stuck state showed the following stack trace:

#0  0x00007fdc23cc9639 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00000000005f15a6 in AbslInternalSpinLockDelay ()
#2  0x00000000004199fc in absl::base_internal::SpinLock::SlowLock() ()
#3  0x0000000000417e3d in tcmalloc::tcmalloc_internal::HugePageAwareAllocator::LockAndAlloc(tcmalloc::tcmalloc_internal::Length, bool*) [clone .cold] ()
#4  0x00000000005d6da6 in tcmalloc::tcmalloc_internal::HugePageAwareAllocator::New(tcmalloc::tcmalloc_internal::Length)
    ()
#5  0x00000000005bab7f in tcmalloc::tcmalloc_internal::CentralFreeList::Populate() ()
#6  0x00000000005bad98 in tcmalloc::tcmalloc_internal::CentralFreeList::RemoveRange(void**, int) ()
#7  0x00000000005be1ae in tcmalloc::tcmalloc_internal::CPUCache::Refill(int, unsigned long) ()
#8  0x00000000005b4f1b in void* tcmalloc::tcmalloc_internal::CPUCache::Allocate<&tcmalloc::tcmalloc_internal::TCMallocPolicy<tcmalloc::tcmalloc_internal::CppOomPolicy, tcmalloc::tcmalloc_internal::DefaultAlignPolicy, tcmalloc::tcmalloc_internal::InvokeHooksPolicy>::handle_oom>(unsigned long)::Helper::Underflow(int, unsigned long) ()
#9  0x0000000000430921 in std::vector<void*, std::allocator<void*> >::_M_default_append(unsigned long) ()