search

facebook / CacheLib

Pluggable in-process caching engine to build and scale high performance services
https://www.cachelib.org
Apache License 2.0
1.22k stars 263 forks source link

Cannot replay requests when amplification factor of requests is set too high #348

Open rainjuns opened 1 week ago

rainjuns commented 1 week ago

Describe the bug

If the amplification factor of requests (i.e., test_config::replayGeneratorConfig::ampFactor) is set too high, KVReplayGenerator stops generating the request and CacheLib continuously shows a hit ratio 0.00% in the output.

I first try to find if the requests are constantly generated and find out that KVReplayGenerator::genRequests() ends up in an infinite loop.

https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/cachebench/workload/KVReplayGenerator.h#L513C1-L519C4

While the consumers (i.e., stressors) have to consume the requests from the queue of stressor (let's say 0), every consumer (i.e., KVReplayGenerator::getReq()) also ends up in an infinite loop.

https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/cachebench/workload/KVReplayGenerator.h#L504C1-L529C2

Because every consumer is stuck in the infinite loop, waiting for requests from queues corresponding to stressors other than 0, the requests in the queue of stressor 0 remain full.

Using GDB and reviewing the source code, I identified three problems.

  1. One of stressor threads keeps waiting in it.wait() in findFn defined in Cache<Allocator>::find(Key key).
  2. One of reader threads is stuck in obtaining TimedMutex so that it cannot finish onGetComplete.
  3. I am not sure this part but, OnGetComplete acquires a lock, hindering removeFromFillMap from acquiring the lock later. So, it seems that lock must be released after checking hasTombStone(hk). https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/allocator/nvmcache/NvmCache.h#L1186C1-L1186C31 https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/allocator/nvmcache/NvmCache.h#L1143C1-L1144C75

Please note that even if (2) is not happening, (1) still happens.

I attached the GDB stacktrace below.

To Reproduce For the fast debugging, I just return Status::NotFound in lookup() function of navy cache.

Here's the JSON file that I used.

{
  "cache_config": {
    "cacheSizeMB": 430,
    "nvmCacheSizeMB": 95230,
    "nvmCachePaths": [],
    "writeAmpDeviceList": [],
    "navyAdmissionWriteRateMB": 50,
    "cacheDir": "/root/cachelib_metadata",
    "allocFactor": 1.08,
    "maxAllocSize": 524288,
    "minAllocSize": 64,
    "navySizeClasses": [],
    "navyReaderThreads": 72,
    "navyWriterThreads": 36,
    "navySmallItemMaxSize": 2048,
    "navySegmentedFifoSegmentRatio": [
      1.0
    ],
    "navyHitsReinsertionThreshold": 1,
    "navyBlockSize": 4096,
    "nvmAdmissionRetentionTimeThreshold": 7200,
    "navyParcelMemoryMB": 6048,
    "enableChainedItem": true,
    "htBucketPower": 29,
    "moveOnSlabRelease": false,
    "poolRebalanceIntervalSec": 2,
    "poolResizeIntervalSec": 2,
    "rebalanceStrategy": "hits"
  },
  "test_config": {
    "traceFileNames": [
      "/root/shared/facebook/kvcache/202206/kvcache_traces_1.csv",
      "/root/shared/facebook/kvcache/202206/kvcache_traces_2.csv"
    ],
    "opRatePerSec": 550000,
    "opRateBurstSize": 200,
    "enableLookaside": false,
    "repeatTraceReplay": false,
    "repeatOpCount": true,
    "onlySetIfMiss": false,
    "numOps": 100000000000,
    "numThreads": 16,
    "prepopulateCache": true,
    "replayGeneratorConfig": {
      "ampFactor": 100
    },
    "generator": "replay"
  }
}

Expected behavior Every consumer thread must not in the infinite loop at a certain point.

Screenshots Here is a part of log messages that I printed for the purpose of debugging.

kv getReq while ... is printed out in the while statement of KVReplayGenerator::getReq(). genReq while ... is printed out in the while statement of KVReplayGenerator::genRequests()

1102 14:18:03.240128 247359 KVReplayGenerator.h:521] kv getReq while, shard id: 12, reqQ empty: true, reqQ full: false
I1102 14:18:03.298928 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:03.298959 247209 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:04.108278 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:04.240181 247357 KVReplayGenerator.h:521] kv getReq while, shard id: 10, reqQ empty: true, reqQ full: false
I1102 14:18:04.298998 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:04.299035 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:05.108232 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:05.240127 247352 KVReplayGenerator.h:521] kv getReq while, shard id: 5, reqQ empty: true, reqQ full: false
I1102 14:18:05.299079 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:05.299106 247232 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:06.108125 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:06.240125 247347 KVReplayGenerator.h:521] kv getReq while, shard id: 0, reqQ empty: true, reqQ full: false
I1102 14:18:06.299164 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:06.299178 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:07.108197 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:07.240201 247349 KVReplayGenerator.h:521] kv getReq while, shard id: 3, reqQ empty: true, reqQ full: false
I1102 14:18:07.299233 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:07.299241 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:08.108180 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:08.240135 247356 KVReplayGenerator.h:521] kv getReq while, shard id: 9, reqQ empty: true, reqQ full: false
I1102 14:18:08.299300 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:08.299303 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:09.108258 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:09.240124 247355 KVReplayGenerator.h:521] kv getReq while, shard id: 8, reqQ empty: true, reqQ full: false
I1102 14:18:09.299386 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:09.299402 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:10.108220 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:10.240131 247351 KVReplayGenerator.h:521] kv getReq while, shard id: 4, reqQ empty: true, reqQ full: false
I1102 14:18:10.299445 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:10.299473 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:11.108255 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:11.240190 247348 KVReplayGenerator.h:521] kv getReq while, shard id: 1, reqQ empty: true, reqQ full: false
I1102 14:18:11.299536 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:11.299564 247232 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:12.108252 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:12.240130 247353 KVReplayGenerator.h:521] kv getReq while, shard id: 6, reqQ empty: true, reqQ full: false
I1102 14:18:12.299608 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:12.299629 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:13.108189 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:13.240127 247352 KVReplayGenerator.h:521] kv getReq while, shard id: 5, reqQ empty: true, reqQ full: false
I1102 14:18:13.299734 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:13.299748 247209 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:14.108222 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:14.240138 247349 KVReplayGenerator.h:521] kv getReq while, shard id: 3, reqQ empty: true, reqQ full: false
I1102 14:18:14.299856 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:14.299855 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:15.108183 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:15.240139 247351 KVReplayGenerator.h:521] kv getReq while, shard id: 4, reqQ empty: true, reqQ full: false
I1102 14:18:15.299937 247209 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:15.299981 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:16.108125 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:16.240120 247352 KVReplayGenerator.h:521] kv getReq while, shard id: 5, reqQ empty: true, reqQ full: false
I1102 14:18:16.300070 247209 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:16.300092 247232 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:17.108263 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:17.240131 247359 KVReplayGenerator.h:521] kv getReq while, shard id: 12, reqQ empty: true, reqQ full: false
I1102 14:18:17.300180 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:17.300220 247209 ThreadPoolJobQueue.cpp:79] running workers: 0

Here is the output of GDB. Certain stressor thread(s) is(are) waiting.

#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff747caff in folly::detail::(anonymous namespace)::nativeFutexWaitImpl (waitMask=4294967295, absSteadyTime=0x0,
    absSystemTime=0x0, expected=4294967293, addr=0x7fe7100bc428) at /root/workspace/kvcache/cachelib/external/folly/folly/detail/Futex.cpp:126
#2  folly::detail::futexWaitImpl (futex=futex@entry=0x7fe7100bc428, expected=expected@entry=4294967293, absSystemTime=absSystemTime@entry=0x0,
    absSteadyTime=absSteadyTime@entry=0x0, waitMask=waitMask@entry=4294967295)
    at /root/workspace/kvcache/cachelib/external/folly/folly/detail/Futex.cpp:254
#3  0x00007ffff74ee6ce in folly::detail::futexWait<std::atomic<unsigned int> > (waitMask=4294967295, expected=4294967293, futex=0x7fe7100bc428)
    at /root/workspace/kvcache/cachelib/external/folly/folly/detail/Futex-inl.h:94
#4  folly::detail::MemoryIdler::futexWait<std::atomic<unsigned int>, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (
    idleTimeout=..., stackToRetain=1024, timeoutVariationFrac=0.5, waitMask=4294967295, expected=4294967293,
    fut=std::atomic<unsigned int> = { 4294967293 }) at /root/workspace/kvcache/cachelib/external/folly/folly/detail/MemoryIdler.h:128
#5  folly::detail::MemoryIdler::futexWait<std::atomic<unsigned int>, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (
    timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, expected=4294967293, fut=...)
    at /root/workspace/kvcache/cachelib/external/folly/folly/detail/MemoryIdler.h:106
#6  folly::fibers::Baton::waitThread (this=0x7fe7100bc428) at /root/workspace/kvcache/cachelib/external/folly/folly/fibers/Baton.cpp:70
#7  0x00007ffff74ee9ae in folly::fibers::Baton::wait<folly::fibers::Baton::wait()::<lambda()> > (mainContextFunc=..., this=<optimized out>)
    at /root/workspace/kvcache/cachelib/external/folly/folly/fibers/Baton-inl.h:54
#8  0x000055555563a14d in facebook::cachelib::detail::ReadHandleImpl<facebook::cachelib::CacheItem<facebook::cachelib::LruCacheTrait> >::ItemWaitContext::wait (this=0x7fe7100bc420) at /root/workspace/kvcache/cachelib/../cachelib/allocator/Handle.h:271
#9  facebook::cachelib::detail::ReadHandleImpl<facebook::cachelib::CacheItem<facebook::cachelib::LruCacheTrait> >::ItemWaitContext::wait (
    this=0x7fe7100bc420) at /root/workspace/kvcache/cachelib/../cachelib/allocator/Handle.h:266
#10 facebook::cachelib::detail::ReadHandleImpl<facebook::cachelib::CacheItem<facebook::cachelib::LruCacheTrait> >::wait (this=0x7fe723ff6450)
    at /root/workspace/kvcache/cachelib/../cachelib/allocator/Handle.h:230
#11 facebook::cachelib::cachebench::Cache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find(facebook::cachelib::KAllocation::Key)::{lambda()#1}::operator()() const (__closure=__closure@entry=0x7fe723ff6260)
    at /root/workspace/kvcache/cachelib/../cachelib/cachebench/cache/Cache.h:1018
#12 0x0000555555648205 in facebook::cachelib::cachebench::Cache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find (
    this=<optimized out>, key=...) at /root/workspace/kvcache/cachelib/../cachelib/cachebench/cache/Cache.h:1036
#13 0x0000555555652763 in facebook::cachelib::cachebench::CacheStressor<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::stressByDiscreteDistribution (this=0x555555bac730, stats=...) at /root/workspace/kvcache/opt/cachelib/include/folly/Range.h:306
#14 0x00007ffff6fb8db4 in std::execute_native_thread_routine (__p=0x7fe71c000b70) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
#15 0x00007ffff6c3fa94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#16 0x00007ffff6cccc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

One of reader threads seems to be waiting to obtain TimeMutex.

#0  0x00007ffff6c8fadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fe756fd4480, rem=0x7fe756fd4480)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007ffff6c9ca27 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00005555556cfbb8 in std::this_thread::sleep_for<long, std::ratio<1l, 1000000000l> > (
    __rtime=<synthetic pointer>std::chrono::duration = { <optimized out>ns }) at /usr/include/c++/13/bits/this_thread_sleep.h:80
#3  folly::detail::Sleeper::wait (this=<optimized out>)
    at /root/workspace/kvcache/opt/cachelib/include/folly/synchronization/detail/Sleeper.h:56
#4  folly::MicroSpinLock::lock (this=0x7fe7142cc821) at /root/workspace/kvcache/opt/cachelib/include/folly/synchronization/MicroSpinLock.h:81
#5  folly::SpinLock::lock (this=0x7fe7142cc821) at /root/workspace/kvcache/opt/cachelib/include/folly/SpinLock.h:45
#6  std::unique_lock<folly::SpinLock>::lock (this=0x7fe756fd4470) at /usr/include/c++/13/bits/unique_lock.h:141
#7  std::unique_lock<folly::SpinLock>::unique_lock (__m=..., this=0x7fe756fd4470) at /usr/include/c++/13/bits/unique_lock.h:71
#8  folly::fibers::TimedMutex::lockHelper<folly::fibers::TimedMutex::lock()::{lambda(folly::fibers::TimedMutex::MutexWaiter&)#1}>(folly::fibers::TimedMutex::lock()::{lambda(folly::fibers::TimedMutex::MutexWaiter&)#1}&&) (this=this@entry=0x7fe7142cc820, waitFunc=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/fibers/TimedMutex-inl.h:30
#9  0x000055555579d510 in folly::fibers::TimedMutex::lock (this=<optimized out>)
    at /root/workspace/kvcache/opt/cachelib/include/folly/fibers/TimedMutex-inl.h:85
#10 std::unique_lock<folly::fibers::TimedMutex>::lock (this=0x7fe756fd4540) at /usr/include/c++/13/bits/unique_lock.h:141
#11 std::unique_lock<folly::fibers::TimedMutex>::unique_lock (__m=..., this=0x7fe756fd4540) at /usr/include/c++/13/bits/unique_lock.h:71
#12 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::getFillLockForShard (shard=7136,
    this=0x7fe7140ed620) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:431
#13 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::getFillLock (hk=...,
    this=<optimized out>) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:435
#14 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::removeFromFillMap (
    this=0x7fe7140ed620, hk=...) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:382
#15 0x00005555557b9bdf in facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}::operator()() const (
    __closure=0x7fe756fd4638) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:1090
#16 folly::catch_exception<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}&, void (*)() noexcept, , void>(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}&, void (*&&)() noexcept) (c=<optimized out>, t=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/lang/Exception.h:290
#17 folly::detail::ScopeGuardImpl<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}, true>::execute() (
    this=0x7fe756fd4630) at /root/workspace/kvcache/opt/cachelib/include/folly/ScopeGuard.h:181
#18 folly::detail::ScopeGuardImpl<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}, true>::~ScopeGuardImpl() (
    this=0x7fe756fd4630, __in_chrg=<optimized out>) at /root/workspace/kvcache/opt/cachelib/include/folly/ScopeGuard.h:154
#19 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete (val=..., hk=...,
    status=<optimized out>, ctx=..., this=<optimized out>) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:1146
#20 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find(facebook::cachelib::HashedKey)::{lambda(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer)#1}::operator()(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer) const (v=..., k=..., s=<optimized out>, __closure=<optimized out>)
    at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:669
#21 folly::detail::function::call_<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find(facebook::cachelib::HashedKey)::{lambda(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer)#1}, true, false, void, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey&&, facebook::cachelib::navy::Buffer&&>(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey&&, facebook::cachelib::navy::Buffer&&, folly::detail::function::Data&) (p=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:341
#22 0x00005555558e054b in folly::detail::function::FunctionTraits<void (facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer)>::operator()(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer) (
    args#2=..., args#1=..., args#0=<optimized out>, this=0x7ffff3b72e10) at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:370
#23 operator() (__closure=0x7ffff3b72e00) at /root/workspace/kvcache/cachelib/navy/engine/EnginePair.cpp:180
#24 0x0000555555879903 in folly::detail::function::FunctionTraits<facebook::cachelib::navy::JobExitCode ()>::operator()() (this=0x7fe710090f50)
    at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:370
#25 operator() (__closure=0x7fe710090f40) at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobScheduler.cpp:192
#26 folly::detail::function::call_<facebook::cachelib::navy::OrderedThreadPoolJobScheduler::scheduleJobLocked(JobParams, uint64_t)::<lambda()>, false, false, facebook::cachelib::navy::JobExitCode>(folly::detail::function::Data &) (p=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:343
#27 0x000055555587d017 in folly::detail::function::FunctionTraits<facebook::cachelib::navy::JobExitCode ()>::operator()() (
    this=<optimized out>) at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:368
#28 operator() (__closure=<optimized out>) at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:158
#29 0x000055555587d5d3 in facebook::cachelib::navy::JobQueue::runJob (this=<optimized out>, entry=...)
    at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:170
#30 0x000055555587d8f3 in facebook::cachelib::navy::JobQueue::process (this=0x55555609d130)
    at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:91
#31 0x00007ffff6fb8db4 in std::execute_native_thread_routine (__p=0x555555be4f50) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
#32 0x00007ffff6c3fa94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
--Type <RET> for more, q to quit, c to continue without paging--
#33 0x00007ffff6cccc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Desktop (please complete the following information):

rainjuns commented 5 days ago

Currently, I am not sure if this problem can be reproduced.