cpu 还能降一降么?

[yourchanges]

Your Environment

• Operating System and version: centos 8.4 64位
• Machine Specifications: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz 56核, 160GB每台 nvme ssd 2TB 每台, 3台 3主3从
• Tendis Version: tendisplus-2.4.3-rocksdb-v5.13.4
• Tendis Configuration:

  bind 0.0.0.0
  port 2000
  daemon yes
  cluster-enabled yes
  requirepass tendis99
  masterauth tendis99
  loglevel notice
  logdir /data1/tendis/tendisplus/log/2000
  dumpdir /data1/tendis/tendisplus/dump/2000
  dir /data1/tendis/tendisplus/db/2000
  pidfile /data1/tendis/tendisplus/tendisplus-2000.pid
  slowlog /data1/tendis/tendisplus/log/2000/slowlog
  rocks.blockcachemb 16384
  rocks.compress_type none  // 或者 lz4
  executorThreadNum 48
  netiothreadnum 12

Description

测试下来, snappy 和 none 压缩算法, cpu都要到2000% , lz4 在1800% 左右, 还有办法调优 降低一点么,

下面是perf top的截图

一主一从 LZ4

Samples: 4M of event 'cycles', 4000 Hz, Event count (approx.): 995508402247 lost: 0/0 drop: 0/419807
Overhead  Shared Object                              Symbol
   2.20%  tendisplus                                 [.] rocksdb::BlockIter::BinarySeek
   1.84%  tendisplus                                 [.] rocksdb::(anonymous namespace)::FullFilterBitsReader::MayMatch
   1.84%  tendisplus                                 [.] rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindGreaterOrEqual
   1.79%  libc-2.28.so                               [.] __memcmp_avx2_movbe
   1.77%  tendisplus                                 [.] rocksdb::Version::Get
   1.63%  [kernel]                                   [k] do_syscall_64
   1.48%  tendisplus                                 [.] free
   1.45%  tendisplus                                 [.] rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
   1.35%  tendisplus                                 [.] rocksdb::InternalKeyComparator::Compare
   1.23%  tendisplus                                 [.] rocksdb::InternalKeyComparator::CompareKeySeq
   1.11%  tendisplus                                 [.] malloc
   1.10%  tendisplus                                 [.] rocksdb::BlockIter::ParseNextKey
   1.09%  tendisplus                                 [.] LZ4_decompress_safe
   1.08%  libpthread-2.28.so                         [.] __pthread_mutex_lock
   1.07%  libc-2.28.so                               [.] __memmove_avx_unaligned_erms
   1.02%  tendisplus                                 [.] rocksdb::MemTable::KeyComparator::operator()
   0.75%  libpthread-2.28.so                         [.] __pthread_mutex_unlock_usercnt
   0.71%  tendisplus                                 [.] tendisplus::NetSession::setResponse
   0.65%  tendisplus                                 [.] rocksdb::Hash
   0.64%  libc-2.28.so                               [.] vfprintf
   0.62%  tendisplus                                 [.] rocksdb::BlockBasedTable::Get
   0.62%  [kernel]                                   [k] switch_mm_irqs_off
   0.62%  [kernel]                                   [k] syscall_return_via_sysret
   0.60%  tendisplus                                 [.] rocksdb::TableCache::Get
   0.57%  [kernel]                                   [k] __schedule
   0.56%  tendisplus                                 [.] rocksdb::Block::NewIterator
   0.54%  [kernel]                                   [k] entry_SYSCALL_64
   0.51%  libc-2.28.so                               [.] sched_getcpu
   0.50%  tendisplus                                 [.] rocksdb::StatisticsImpl::recordTick
   0.50%  [kernel]                                   [k] copy_user_enhanced_fast_string
   0.48%  [kernel]                                   [k] menu_select
   0.47%  tendisplus                                 [.] rocksdb::DBImpl::GetImpl
   0.46%  tendisplus                                 [.] std::vector<unsigned char, std::allocator<unsigned char> >::_M_range_insert<__gnu_cxx::__normal_iterator<char const*
   0.44%  [kernel]                                   [k] _raw_spin_lock_irqsave
   0.41%  tendisplus                                 [.] rocksdb::LRUHandleTable::FindPointer
   0.39%  [kernel]                                   [k] _raw_spin_lock
   0.38%  tendisplus                                 [.] rocksdb::HistogramBucketMapper::IndexForValue
   0.35%  [kernel]                                   [k] futex_wake
Too slow to read ring buffer (change period (-c/-F) or limit CPUs (-C)

一主一从 none 压缩

Samples: 894K of event 'cycles', 4000 Hz, Event count (approx.): 401056174274 lost: 0/0 drop: 0/459196
Overhead  Shared Object                             Symbol
   2.79%  tendisplus                                [.] snappy::RawUncompress
   2.45%  tendisplus                                [.] rocksdb::BlockIter::BinarySeek
   2.08%  tendisplus                                [.] rocksdb::(anonymous namespace)::FullFilterBitsReader::MayMatch
   2.00%  tendisplus                                [.] rocksdb::Version::Get
   1.90%  libc-2.28.so                              [.] __memcmp_avx2_movbe
   1.87%  tendisplus                                [.] rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindGreaterOrEqual
   1.59%  [kernel]                                  [k] do_syscall_64
   1.55%  tendisplus                                [.] rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
   1.54%  tendisplus                                [.] free
   1.45%  tendisplus                                [.] rocksdb::InternalKeyComparator::Compare
   1.29%  tendisplus                                [.] rocksdb::InternalKeyComparator::CompareKeySeq
   1.20%  tendisplus                                [.] malloc
   1.17%  tendisplus                                [.] rocksdb::BlockIter::ParseNextKey
   1.15%  libpthread-2.28.so                        [.] __pthread_mutex_lock
   1.07%  tendisplus                                [.] rocksdb::MemTable::KeyComparator::operator()
   1.06%  libc-2.28.so                              [.] __memmove_avx_unaligned_erms
   0.81%  libpthread-2.28.so                        [.] __pthread_mutex_unlock_usercnt
   0.77%  tendisplus                                [.] rocksdb::TableCache::Get
   0.76%  tendisplus                                [.] tendisplus::NetSession::setResponse
   0.70%  tendisplus                                [.] rocksdb::BlockBasedTable::Get
   0.68%  tendisplus                                [.] rocksdb::Hash
   0.64%  libc-2.28.so                              [.] vfprintf
   0.62%  [kernel]                                  [k] syscall_return_via_sysret
   0.58%  [kernel]                                  [k] switch_mm_irqs_off
   0.57%  libc-2.28.so                              [.] sched_getcpu
   0.56%  [kernel]                                  [k] __schedule
   0.56%  tendisplus                                [.] rocksdb::StatisticsImpl::recordTick
   0.55%  tendisplus                                [.] rocksdb::Block::NewIterator
   0.54%  [kernel]                                  [k] copy_user_enhanced_fast_string
   0.53%  [kernel]                                  [k] update_load_avg
   0.52%  [kernel]                                  [k] entry_SYSCALL_64
   0.49%  [kernel]                                  [k] menu_select
   0.47%  tendisplus                                [.] rocksdb::DBImpl::GetImpl
   0.47%  tendisplus                                [.] rocksdb::LRUHandleTable::FindPointer
   0.46%  tendisplus                                [.] rocksdb::HistogramBucketMapper::IndexForValue
   0.44%  [kernel]                                  [k] _raw_spin_lock_irqsave
   0.42%  tendisplus                                [.] std::vector<unsigned char, std::allocator<unsigned char> >::_M_range_insert<__gnu_cxx::__normal_iterator<char const*,
   0.42%  [vdso]                                    [.] __vdso_getcpu
Too slow to read ring buffer (change period (-c/-F) or limit CPUs (-C)

最后一主一从 lz4

Samples: 856K of event 'cycles', 4000 Hz, Event count (approx.): 372364926233 lost: 0/0 drop: 0/458227
Overhead  Shared Object                             Symbol
   2.61%  tendisplus                                [.] rocksdb::BlockIter::BinarySeek
   2.22%  tendisplus                                [.] rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>::FindGreaterOrEqual
   2.02%  tendisplus                                [.] rocksdb::Version::Get
   2.01%  libc-2.28.so                              [.] __memcmp_avx2_movbe
   1.86%  tendisplus                                [.] LZ4_decompress_safe
   1.84%  tendisplus                                [.] rocksdb::(anonymous namespace)::FullFilterBitsReader::MayMatch
   1.68%  [kernel]                                  [k] do_syscall_64
   1.65%  tendisplus                                [.] rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
   1.56%  tendisplus                                [.] rocksdb::InternalKeyComparator::Compare
   1.48%  tendisplus                                [.] free
   1.41%  tendisplus                                [.] rocksdb::InternalKeyComparator::CompareKeySeq
   1.21%  libpthread-2.28.so                        [.] __pthread_mutex_lock
   1.21%  tendisplus                                [.] rocksdb::BlockIter::ParseNextKey
   1.20%  tendisplus                                [.] rocksdb::MemTable::KeyComparator::operator()
   1.18%  tendisplus                                [.] malloc
   1.15%  libjvm.so                                 [.] SpinPause
   0.90%  libc-2.28.so                              [.] __memmove_avx_unaligned_erms
   0.84%  libpthread-2.28.so                        [.] __pthread_mutex_unlock_usercnt
   0.83%  tendisplus                                [.] tendisplus::NetSession::setResponse
   0.70%  tendisplus                                [.] rocksdb::BlockBasedTable::Get
   0.69%  tendisplus                                [.] rocksdb::TableCache::Get
   0.65%  [kernel]                                  [k] switch_mm_irqs_off
   0.64%  [kernel]                                  [k] syscall_return_via_sysret
   0.64%  tendisplus                                [.] rocksdb::Block::NewIterator
   0.63%  [kernel]                                  [k] copy_user_enhanced_fast_string
   0.63%  libc-2.28.so                              [.] vfprintf
   0.60%  tendisplus                                [.] rocksdb::Hash
   0.59%  libc-2.28.so                              [.] sched_getcpu
   0.56%  tendisplus                                [.] rocksdb::StatisticsImpl::recordTick
   0.55%  [kernel]                                  [k] entry_SYSCALL_64
   0.54%  [kernel]                                  [k] __schedule
   0.52%  tendisplus                                [.] rocksdb::DBImpl::GetImpl
   0.50%  [kernel]                                  [k] menu_select
   0.50%  libc-2.28.so                              [.] __memmove_sse2_unaligned_erms
   0.50%  liblz4-java3672842715170129947.so         [.] LZ4_decompress_fast
   0.47%  tendisplus                                [.] rocksdb::LRUHandleTable::FindPointer
   0.46%  tendisplus                                [.] rocksdb::HistogramBucketMapper::IndexForValue
   0.45%  [kernel]                                  [k] _raw_spin_lock_irqsave
Too slow to read ring buffer (change period (-c/-F) or limit CPUs (-C)

另外 就是 配置的16GB 实际占用的是22GB, 6个节点就是120GB左右了 总共

[raffertyyu]

后续版本会考虑优化CPU和内存方面的占用。感谢反馈，敬请期待。

[yourchanges]

请问有进展么

[takenliu]

这个cpu使用率2000%是正常的哦，关键还是看你压测的qps有多大。一般qps到了20万级别的话，cpu利用率就很容易到2000%的。 内存使用方面，参考我在这个issue里面的回复：https://github.com/Tencent/Tendis/issues/167 https://github.com/Tencent/Tendis/issues/167#issuecomment-1192148757