search

sofastack / sofa-jraft

A production-grade java implementation of RAFT consensus algorithm.
https://www.sofastack.tech/projects/sofa-jraft/
Apache License 2.0
3.57k stars 1.14k forks source link

nacos的RingBuffer内存泄露 #948

Closed zz630 closed 1 year ago

zz630 commented 1 year ago

Your question

MAT分析RingBuffer占了1.6g，fullgc后不下降，已参考#723和#833 nacos的写qps不高，-Djraft.recyclers.maxCapacityPerThread=0关了recycle。disruptorBufferSize是默认的16384

Environment

zz630 commented 1 year ago

node_describe：

nodeId: <naming_service_metadata/164.90.68.130:14432> state: STATE_FOLLOWER leaderId: 10.235.191.6:14432 term: 24 conf: ConfigurationEntry [id=LogId [index=93, term=24], conf=10.235.191.5:14432,10.235.191.6:14432,10.236.191.5:14432,169.136.78.33:14432,164.90.68.130:14432, oldConf=] targetPriority: -1 electionTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@2de823a7, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-ElectionTimer-<naming_service_metadata/164.90.68.130:14432>'} voteTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-VoteTimer-<naming_service_metadata/164.90.68.130:14432>'} stepDownTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=2500, name='JRaft-StepDownTimer-<naming_service_metadata/164.90.68.130:14432>'} snapshotTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@68eaa87e, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=1800000, name='JRaft-SnapshotTimer-<naming_service_metadata/164.90.68.130:14432>'} logManager: storage: [94, 93] diskId: LogId [index=93, term=24] appliedId: LogId [index=93, term=24] lastSnapshotId: LogId [index=93, term=24] fsmCaller: StateMachine [Idle] ballotBox: lastCommittedIndex: 93 pendingIndex: 0 pendingMetaQueueSize: 0 snapshotExecutor: lastSnapshotTerm: 24 lastSnapshotIndex: 93 term: 24 savingSnapshot: false loadingSnapshot: false stopped: false replicatorGroup: replicators: [] failureReplicators: {} logStorage:

Compaction Stats [default] Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop

L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 1 0.023 0 0 L1 1/0 0.87 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.1 0.1 0 1 0.016 1 0 Sum 1/0 0.87 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.9 0.0 0.0 0 2 0.020 1 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 6074950.3 total, 518.5 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

File Read Latency Histogram By Level [default] ** Level 0 read latency histogram (micros): Count: 13 Average: 3.3077 StdDev: 3.99 Min: 0 Median: 0.8125 Max: 14 Percentiles: P50: 0.81 P75: 4.75 P99: 14.00 P99.9: 14.00 P99.99: 14.00

[ 0, 1 ] 8 61.538% 61.538% ############ ( 1, 2 ] 1 7.692% 69.231% ## ( 4, 6 ] 2 15.385% 84.615% ### ( 6, 10 ] 1 7.692% 92.308% ## ( 10, 15 ] 1 7.692% 100.000% ##

** Level 1 read latency histogram (micros): Count: 6 Average: 4.8333 StdDev: 4.71 Min: 1 Median: 1.6667 Max: 14 Percentiles: P50: 1.67 P75: 8.00 P99: 14.00 P99.9: 14.00 P99.99: 14.00

[ 0, 1 ] 1 16.667% 16.667% ### ( 1, 2 ] 3 50.000% 66.667% ########## ( 6, 10 ] 1 16.667% 83.333% ### ( 10, 15 ] 1 16.667% 100.000% ###

DB Stats Uptime(secs): 6074950.3 total, 518.5 interval Cumulative writes: 10K writes, 10K keys, 10K commit groups, 1.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 10K writes, 3379 syncs, 3.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent

rocksdb.block.cache.miss COUNT : 14 rocksdb.block.cache.hit COUNT : 21 rocksdb.block.cache.add COUNT : 13 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 5 rocksdb.block.cache.index.hit COUNT : 13 rocksdb.block.cache.index.add COUNT : 5 rocksdb.block.cache.index.bytes.insert COUNT : 672 rocksdb.block.cache.index.bytes.evict COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.filter.bytes.evict COUNT : 0 rocksdb.block.cache.data.miss COUNT : 9 rocksdb.block.cache.data.hit COUNT : 8 rocksdb.block.cache.data.add COUNT : 8 rocksdb.block.cache.data.bytes.insert COUNT : 1824 rocksdb.block.cache.bytes.read COUNT : 3648 rocksdb.block.cache.bytes.write COUNT : 2496 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 0 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 9024 rocksdb.memtable.miss COUNT : 7 rocksdb.l0.hit COUNT : 2 rocksdb.l1.hit COUNT : 5 rocksdb.l2andup.hit COUNT : 0 rocksdb.compaction.key.drop.new COUNT : 1 rocksdb.compaction.key.drop.obsolete COUNT : 16 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 16 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 10136 rocksdb.number.keys.read COUNT : 9031 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 355344 rocksdb.bytes.read COUNT : 1047596 rocksdb.number.db.seek COUNT : 2 rocksdb.number.db.next COUNT : 2 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 2 rocksdb.number.db.next.found COUNT : 1 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 274 rocksdb.no.file.closes COUNT : 0 rocksdb.no.file.opens COUNT : 6 rocksdb.no.file.errors COUNT : 0 rocksdb.l0.slowdown.micros COUNT : 0 rocksdb.memtable.compaction.micros COUNT : 0 rocksdb.l0.num.files.stall.micros COUNT : 0 rocksdb.stall.micros COUNT : 0 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.rate.limit.delay.millis COUNT : 0 rocksdb.num.iterators COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.deletes.filtered COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.block.cachecompressed.miss COUNT : 0 rocksdb.block.cachecompressed.hit COUNT : 0 rocksdb.block.cachecompressed.add COUNT : 0 rocksdb.block.cachecompressed.add.failures COUNT : 0 rocksdb.wal.synced COUNT : 3379 rocksdb.wal.bytes COUNT : 355344 rocksdb.write.self COUNT : 10133 rocksdb.write.other COUNT : 0 rocksdb.write.timeout COUNT : 0 rocksdb.write.wal COUNT : 20266 rocksdb.compact.read.bytes COUNT : 2612 rocksdb.compact.write.bytes COUNT : 3773 rocksdb.flush.write.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 0 rocksdb.number.superversion_acquires COUNT : 21 rocksdb.number.superversion_releases COUNT : 0 rocksdb.number.superversion_cleanups COUNT : 0 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 1 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.overwritten COUNT : 0 rocksdb.blobdb.gc.num.keys.expired COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.overwritten COUNT : 0 rocksdb.blobdb.gc.bytes.expired COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 2 rocksdb.num.iterator.deleted COUNT : 2 rocksdb.db.get.micros P50 : 59.077428 P95 : 2752.372014 P99 : 2875.662116 P100 : 3762.000000 COUNT : 9031 SUM : 7221517 rocksdb.db.write.micros P50 : 61.183883 P95 : 35822.071130 P99 : 61116.666667 P100 : 344142.000000 COUNT : 10133 SUM : 86382236 rocksdb.compaction.times.micros P50 : 22000.000000 P95 : 40010.000000 P99 : 40010.000000 P100 : 40010.000000 COUNT : 2 SUM : 56366 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 18000.000000 P95 : 21506.000000 P99 : 21506.000000 P100 : 21506.000000 COUNT : 2 SUM : 38171 rocksdb.compaction.outfile.sync.micros P50 : 22000.000000 P95 : 39510.000000 P99 : 39510.000000 P100 : 39510.000000 COUNT : 2 SUM : 55192 rocksdb.wal.file.sync.micros P50 : 19106.992418 P95 : 47678.633406 P99 : 192200.000000 P100 : 344066.000000 COUNT : 3379 SUM : 84661584 rocksdb.manifest.file.sync.micros P50 : 22000.000000 P95 : 45569.000000 P99 : 45569.000000 P100 : 45569.000000 COUNT : 4 SUM : 115290 rocksdb.table.open.io.micros P50 : 125.000000 P95 : 141.000000 P99 : 141.000000 P100 : 141.000000 COUNT : 6 SUM : 691 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.get.micros P50 : 1.750000 P95 : 25.600000 P99 : 31.000000 P100 : 31.000000 COUNT : 14 SUM : 91 rocksdb.write.raw.block.micros P50 : 0.647059 P95 : 21.300000 P99 : 30.000000 P100 : 30.000000 COUNT : 22 SUM : 69 rocksdb.l0.slowdown.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.memtable.compaction.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.files.stall.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.hard.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.soft.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.numfiles.in.singlecompaction P50 : 2.000000 P95 : 2.000000 P99 : 2.000000 P100 : 2.000000 COUNT : 2 SUM : 4 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.sst.read.micros P50 : 0.863636 P95 : 12.750000 P99 : 19.340000 P100 : 20.000000 COUNT : 38 SUM : 125 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 116.000000 P95 : 116.000000 P99 : 116.000000 P100 : 116.000000 COUNT : 9031 SUM : 1047596 rocksdb.bytes.per.write P50 : 31.001777 P95 : 48.463848 P99 : 50.504855 P100 : 267.000000 COUNT : 10133 SUM : 355344 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.gc.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0

nodeId: <naming_persistent_service/164.90.68.130:14432> state: STATE_FOLLOWER leaderId: 10.236.191.5:14432 term: 24 conf: ConfigurationEntry [id=LogId [index=3352135, term=24], conf=10.235.191.5:14432,10.235.191.6:14432,10.236.191.5:14432,169.136.78.33:14432,164.90.68.130:14432, oldConf=] targetPriority: -1 electionTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@4d990344, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-ElectionTimer-<naming_persistent_service/164.90.68.130:14432>'} voteTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-VoteTimer-<naming_persistent_service/164.90.68.130:14432>'} stepDownTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=2500, name='JRaft-StepDownTimer-<naming_persistent_service/164.90.68.130:14432>'} snapshotTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@59c84538, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=1800000, name='JRaft-SnapshotTimer-<naming_persistent_service/164.90.68.130:14432>'} logManager: storage: [3352103, 3352156] diskId: LogId [index=3352156, term=24] appliedId: LogId [index=3352156, term=24] lastSnapshotId: LogId [index=3352135, term=24] fsmCaller: StateMachine [Idle] ballotBox: lastCommittedIndex: 3352156 pendingIndex: 0 pendingMetaQueueSize: 0 snapshotExecutor: lastSnapshotTerm: 24 lastSnapshotIndex: 3352135 term: 24 savingSnapshot: false loadingSnapshot: false stopped: false replicatorGroup: replicators: [] failureReplicators: {} logStorage:

Compaction Stats [default] Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop

L0 1/0 2.57 MB 0.5 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 54.2 3 32 0.082 0 0 L1 1/0 2.74 MB 0.0 0.2 0.1 0.1 0.1 -0.0 0.0 0.5 87.4 29.5 3 16 0.168 23K 18K L2 6/0 228.16 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Sum 8/0 233.48 MB 0.0 0.2 0.1 0.1 0.2 0.1 0.0 1.6 44.0 41.8 5 48 0.111 23K 18K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 6074989.7 total, 544.3 interval Flush(GB): cumulative 0.140, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.22 GB write, 0.00 MB/s write, 0.23 GB read, 0.00 MB/s read, 5.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

File Read Latency Histogram By Level [default] ** Level 0 read latency histogram (micros): Count: 4177 Average: 144.1010 StdDev: 942.56 Min: 0 Median: 2.1151 Max: 20680 Percentiles: P50: 2.12 P75: 3.93 P99: 2622.50 P99.9: 13463.81 P99.99: 20680.00

[ 0, 1 ] 921 22.049% 22.049% #### ( 1, 2 ] 1076 25.760% 47.809% ##### ( 2, 3 ] 795 19.033% 66.842% #### ( 3, 4 ] 367 8.786% 75.628% ## ( 4, 6 ] 328 7.853% 83.481% ## ( 6, 10 ] 179 4.285% 87.766% # ( 10, 15 ] 30 0.718% 88.485% ( 15, 22 ] 9 0.215% 88.700% ( 22, 34 ] 9 0.215% 88.915% ( 34, 51 ] 47 1.125% 90.041% ( 51, 76 ] 75 1.796% 91.836% ( 76, 110 ] 57 1.365% 93.201% ( 110, 170 ] 24 0.575% 93.775% ( 170, 250 ] 5 0.120% 93.895% ( 250, 380 ] 4 0.096% 93.991% ( 380, 580 ] 52 1.245% 95.236% ( 580, 870 ] 20 0.479% 95.715% ( 870, 1300 ] 42 1.006% 96.720% ( 1300, 1900 ] 75 1.796% 98.516% ( 1900, 2900 ] 28 0.670% 99.186% ( 2900, 4400 ] 1 0.024% 99.210% ( 4400, 6600 ] 8 0.192% 99.401% ( 6600, 9900 ] 13 0.311% 99.713% ( 9900, 14000 ] 9 0.215% 99.928% ( 14000, 22000 ] 3 0.072% 100.000%

** Level 1 read latency histogram (micros): Count: 2567 Average: 230.4250 StdDev: 1346.23 Min: 0 Median: 2.9792 Max: 34834 Percentiles: P50: 2.98 P75: 5.89 P99: 2827.84 P99.9: 19492.80 P99.99: 34834.00

[ 0, 1 ] 505 19.673% 19.673% #### ( 1, 2 ] 520 20.257% 39.930% #### ( 2, 3 ] 264 10.284% 50.214% ## ( 3, 4 ] 268 10.440% 60.654% ## ( 4, 6 ] 390 15.193% 75.847% ### ( 6, 10 ] 187 7.285% 83.132% # ( 10, 15 ] 36 1.402% 84.534% ( 15, 22 ] 23 0.896% 85.430% ( 22, 34 ] 17 0.662% 86.093% ( 34, 51 ] 5 0.195% 86.287% ( 76, 110 ] 28 1.091% 87.378% ( 110, 170 ] 5 0.195% 87.573% ( 170, 250 ] 7 0.273% 87.846% ( 250, 380 ] 18 0.701% 88.547% ( 380, 580 ] 57 2.220% 90.767% ( 580, 870 ] 58 2.259% 93.027% ( 870, 1300 ] 56 2.182% 95.208% ( 1300, 1900 ] 63 2.454% 97.663% ( 1900, 2900 ] 37 1.441% 99.104% ( 2900, 4400 ] 2 0.078% 99.182% ( 4400, 6600 ] 4 0.156% 99.338% ( 6600, 9900 ] 6 0.234% 99.571% ( 9900, 14000 ] 5 0.195% 99.766% ( 14000, 22000 ] 5 0.195% 99.961% ( 33000, 50000 ] 1 0.039% 100.000%

** Level 2 read latency histogram (micros): Count: 26 Average: 260.6923 StdDev: 1284.87 Min: 0 Median: 0.8125 Max: 6685 Percentiles: P50: 0.81 P75: 6.50 P99: 6685.00 P99.9: 6685.00 P99.99: 6685.00

[ 0, 1 ] 16 61.538% 61.538% ############ ( 2, 3 ] 1 3.846% 65.385% # ( 3, 4 ] 1 3.846% 69.231% # ( 4, 6 ] 1 3.846% 73.077% # ( 6, 10 ] 4 15.385% 88.462% ### ( 10, 15 ] 2 7.692% 96.154% ## ( 6600, 9900 ] 1 3.846% 100.000% #

DB Stats Uptime(secs): 6074989.7 total, 544.3 interval Cumulative writes: 299K writes, 309K keys, 299K commit groups, 1.0 writes per commit group, ingest: 6.30 GB, 0.00 MB/s Cumulative WAL: 299K writes, 292K syncs, 1.02 writes per sync, written: 6.30 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 29 writes, 29 keys, 29 commit groups, 1.0 writes per commit group, ingest: 1.54 MB, 0.00 MB/s Interval WAL: 29 writes, 27 syncs, 1.04 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent

rocksdb.block.cache.miss COUNT : 6564 rocksdb.block.cache.hit COUNT : 25204 rocksdb.block.cache.add COUNT : 1282 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 67 rocksdb.block.cache.index.hit COUNT : 12632 rocksdb.block.cache.index.add COUNT : 66 rocksdb.block.cache.index.bytes.insert COUNT : 205856 rocksdb.block.cache.index.bytes.evict COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.filter.bytes.evict COUNT : 0 rocksdb.block.cache.data.miss COUNT : 6497 rocksdb.block.cache.data.hit COUNT : 12572 rocksdb.block.cache.data.add COUNT : 1216 rocksdb.block.cache.data.bytes.insert COUNT : 11247040 rocksdb.block.cache.bytes.read COUNT : 148612368 rocksdb.block.cache.bytes.write COUNT : 11452896 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 0 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 40823414 rocksdb.memtable.miss COUNT : 12568 rocksdb.l0.hit COUNT : 11634 rocksdb.l1.hit COUNT : 934 rocksdb.l2andup.hit COUNT : 0 rocksdb.compaction.key.drop.new COUNT : 18910 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 18910 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 309174 rocksdb.number.keys.read COUNT : 40835982 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 6759468339 rocksdb.bytes.read COUNT : 2729444360315 rocksdb.number.db.seek COUNT : 2 rocksdb.number.db.next COUNT : 1 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 2 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 204 rocksdb.no.file.closes COUNT : 0 rocksdb.no.file.opens COUNT : 58 rocksdb.no.file.errors COUNT : 0 rocksdb.l0.slowdown.micros COUNT : 0 rocksdb.memtable.compaction.micros COUNT : 0 rocksdb.l0.num.files.stall.micros COUNT : 0 rocksdb.stall.micros COUNT : 0 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.rate.limit.delay.millis COUNT : 0 rocksdb.num.iterators COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.deletes.filtered COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.block.cachecompressed.miss COUNT : 0 rocksdb.block.cachecompressed.hit COUNT : 0 rocksdb.block.cachecompressed.add COUNT : 0 rocksdb.block.cachecompressed.add.failures COUNT : 0 rocksdb.wal.synced COUNT : 292712 rocksdb.wal.bytes COUNT : 6759468339 rocksdb.write.self COUNT : 299464 rocksdb.write.other COUNT : 0 rocksdb.write.timeout COUNT : 0 rocksdb.write.wal COUNT : 598928 rocksdb.compact.read.bytes COUNT : 235690531 rocksdb.compact.write.bytes COUNT : 96024260 rocksdb.flush.write.bytes COUNT : 136859726 rocksdb.number.direct.load.table.properties COUNT : 0 rocksdb.number.superversion_acquires COUNT : 1914 rocksdb.number.superversion_releases COUNT : 0 rocksdb.number.superversion_cleanups COUNT : 0 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 8 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 1 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.overwritten COUNT : 0 rocksdb.blobdb.gc.num.keys.expired COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.overwritten COUNT : 0 rocksdb.blobdb.gc.bytes.expired COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 2 rocksdb.num.iterator.deleted COUNT : 2 rocksdb.db.get.micros P50 : 54.585046 P95 : 156.744487 P99 : 326.810228 P100 : 9664.000000 COUNT : 40835982 SUM : 2715711784 rocksdb.db.write.micros P50 : 19645.280632 P95 : 45567.420290 P99 : 153867.826087 P100 : 3531889.000000 COUNT : 299464 SUM : 7197968429 rocksdb.compaction.times.micros P50 : 158000.000000 P95 : 252971.000000 P99 : 252971.000000 P100 : 252971.000000 COUNT : 16 SUM : 2680356 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 49392.857143 P95 : 87250.000000 P99 : 116649.000000 P100 : 116649.000000 COUNT : 33 SUM : 1761295 rocksdb.compaction.outfile.sync.micros P50 : 50000.000000 P95 : 100666.666667 P99 : 102606.000000 P100 : 102606.000000 COUNT : 16 SUM : 942561 rocksdb.wal.file.sync.micros P50 : 19676.961794 P95 : 45711.319914 P99 : 155368.133536 P100 : 3531766.000000 COUNT : 292712 SUM : 7151932971 rocksdb.manifest.file.sync.micros P50 : 18292.682927 P95 : 31533.333333 P99 : 40097.000000 P100 : 40097.000000 COUNT : 48 SUM : 883200 rocksdb.table.open.io.micros P50 : 237.692308 P95 : 6710.000000 P99 : 9262.000000 P100 : 9532.000000 COUNT : 58 SUM : 59358 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.get.micros P50 : 14.618529 P95 : 1361.666667 P99 : 2839.154930 P100 : 35156.000000 COUNT : 6564 SUM : 1493014 rocksdb.write.raw.block.micros P50 : 9.082749 P95 : 554.035088 P99 : 1152.694972 P100 : 1320.000000 COUNT : 6232 SUM : 468392 rocksdb.l0.slowdown.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.memtable.compaction.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.files.stall.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.hard.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.soft.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.numfiles.in.singlecompaction P50 : 2.000000 P95 : 2.000000 P99 : 2.000000 P100 : 2.000000 COUNT : 16 SUM : 32 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.sst.read.micros P50 : 2.325636 P95 : 957.096939 P99 : 2748.769231 P100 : 34834.000000 COUNT : 6783 SUM : 1200229 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1211.305294 P95 : 347889.912556 P99 : 413840.000000 P100 : 413840.000000 COUNT : 40835982 SUM : 2729444360315 rocksdb.bytes.per.write P50 : 1122.973869 P95 : 256456.224202 P99 : 369224.187124 P100 : 529268.000000 COUNT : 299464 SUM : 6759468339 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 1085.000000 P95 : 218000.000000 P99 : 219849.000000 P100 : 219849.000000 COUNT : 8 SUM : 232922 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.gc.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 71666.666667 P95 : 125000.000000 P99 : 159945.000000 P100 : 159945.000000 COUNT : 30 SUM : 2469613

nodeId: <naming_instance_metadata/164.90.68.130:14432> state: STATE_FOLLOWER leaderId: 10.235.191.6:14432 term: 24 conf: ConfigurationEntry [id=LogId [index=44, term=24], conf=10.235.191.5:14432,10.235.191.6:14432,10.236.191.5:14432,169.136.78.33:14432,164.90.68.130:14432, oldConf=] targetPriority: -1 electionTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@3551b0bd, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-ElectionTimer-<naming_instance_metadata/164.90.68.130:14432>'} voteTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-VoteTimer-<naming_instance_metadata/164.90.68.130:14432>'} stepDownTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=2500, name='JRaft-StepDownTimer-<naming_instance_metadata/164.90.68.130:14432>'} snapshotTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@8c55731, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=1800000, name='JRaft-SnapshotTimer-<naming_instance_metadata/164.90.68.130:14432>'} logManager: storage: [45, 44] diskId: LogId [index=44, term=24] appliedId: LogId [index=44, term=24] lastSnapshotId: LogId [index=44, term=24] fsmCaller: StateMachine [Idle] ballotBox: lastCommittedIndex: 44 pendingIndex: 0 pendingMetaQueueSize: 0 snapshotExecutor: lastSnapshotTerm: 24 lastSnapshotIndex: 44 term: 24 savingSnapshot: false loadingSnapshot: false stopped: false replicatorGroup: replicators: [] failureReplicators: {} logStorage:

Compaction Stats [default] Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop

L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 1 0.038 0 0 L1 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0 1 0.023 0 0 Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 2 0.031 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 6074950.1 total, 522.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.1 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

File Read Latency Histogram By Level [default] ** Level 0 read latency histogram (micros): Count: 10 Average: 3.5000 StdDev: 4.30 Min: 1 Median: 1.0000 Max: 13 Percentiles: P50: 1.00 P75: 1.83 P99: 13.00 P99.9: 13.00 P99.99: 13.00

[ 0, 1 ] 5 50.000% 50.000% ########## ( 1, 2 ] 3 30.000% 80.000% ###### ( 10, 15 ] 2 20.000% 100.000% ####

DB Stats Uptime(secs): 6074950.1 total, 522.0 interval Cumulative writes: 10K writes, 10K keys, 10K commit groups, 1.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 10K writes, 3379 syncs, 3.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent

rocksdb.block.cache.miss COUNT : 9 rocksdb.block.cache.hit COUNT : 6 rocksdb.block.cache.add COUNT : 9 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 3 rocksdb.block.cache.index.hit COUNT : 4 rocksdb.block.cache.index.add COUNT : 3 rocksdb.block.cache.index.bytes.insert COUNT : 416 rocksdb.block.cache.index.bytes.evict COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.filter.bytes.evict COUNT : 0 rocksdb.block.cache.data.miss COUNT : 6 rocksdb.block.cache.data.hit COUNT : 2 rocksdb.block.cache.data.add COUNT : 6 rocksdb.block.cache.data.bytes.insert COUNT : 1104 rocksdb.block.cache.bytes.read COUNT : 848 rocksdb.block.cache.bytes.write COUNT : 1520 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 0 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 14761 rocksdb.memtable.miss COUNT : 0 rocksdb.l0.hit COUNT : 0 rocksdb.l1.hit COUNT : 0 rocksdb.l2andup.hit COUNT : 0 rocksdb.compaction.key.drop.new COUNT : 1 rocksdb.compaction.key.drop.obsolete COUNT : 12 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 12 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 10135 rocksdb.number.keys.read COUNT : 14761 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 355506 rocksdb.bytes.read COUNT : 1712276 rocksdb.number.db.seek COUNT : 2 rocksdb.number.db.next COUNT : 1 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 1 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 26 rocksdb.no.file.closes COUNT : 0 rocksdb.no.file.opens COUNT : 5 rocksdb.no.file.errors COUNT : 0 rocksdb.l0.slowdown.micros COUNT : 0 rocksdb.memtable.compaction.micros COUNT : 0 rocksdb.l0.num.files.stall.micros COUNT : 0 rocksdb.stall.micros COUNT : 0 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.rate.limit.delay.millis COUNT : 0 rocksdb.num.iterators COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.deletes.filtered COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.block.cachecompressed.miss COUNT : 0 rocksdb.block.cachecompressed.hit COUNT : 0 rocksdb.block.cachecompressed.add COUNT : 0 rocksdb.block.cachecompressed.add.failures COUNT : 0 rocksdb.wal.synced COUNT : 3379 rocksdb.wal.bytes COUNT : 355506 rocksdb.write.self COUNT : 10131 rocksdb.write.other COUNT : 0 rocksdb.write.timeout COUNT : 0 rocksdb.write.wal COUNT : 20262 rocksdb.compact.read.bytes COUNT : 1232 rocksdb.compact.write.bytes COUNT : 3093 rocksdb.flush.write.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 0 rocksdb.number.superversion_acquires COUNT : 18 rocksdb.number.superversion_releases COUNT : 0 rocksdb.number.superversion_cleanups COUNT : 0 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 1 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.overwritten COUNT : 0 rocksdb.blobdb.gc.num.keys.expired COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.overwritten COUNT : 0 rocksdb.blobdb.gc.bytes.expired COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 2 rocksdb.num.iterator.deleted COUNT : 2 rocksdb.db.get.micros P50 : 64.879284 P95 : 2683.308428 P99 : 2859.769874 P100 : 3412.000000 COUNT : 14761 SUM : 11298330 rocksdb.db.write.micros P50 : 63.798295 P95 : 28733.383234 P99 : 47691.943574 P100 : 276555.000000 COUNT : 10131 SUM : 75584480 rocksdb.compaction.times.micros P50 : 22000.000000 P95 : 23379.000000 P99 : 23379.000000 P100 : 23379.000000 COUNT : 2 SUM : 40530 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 22000.000000 P95 : 37435.000000 P99 : 37435.000000 P100 : 37435.000000 COUNT : 2 SUM : 53924 rocksdb.compaction.outfile.sync.micros P50 : 22000.000000 P95 : 22975.000000 P99 : 22975.000000 P100 : 22975.000000 COUNT : 2 SUM : 39726 rocksdb.wal.file.sync.micros P50 : 18696.449026 P95 : 43820.941558 P99 : 104255.882353 P100 : 276435.000000 COUNT : 3379 SUM : 74064496 rocksdb.manifest.file.sync.micros P50 : 22000.000000 P95 : 36567.000000 P99 : 36567.000000 P100 : 36567.000000 COUNT : 4 SUM : 105500 rocksdb.table.open.io.micros P50 : 125.000000 P95 : 174.000000 P99 : 174.000000 P100 : 174.000000 COUNT : 5 SUM : 637 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.get.micros P50 : 2.250000 P95 : 16.000000 P99 : 16.000000 P100 : 16.000000 COUNT : 9 SUM : 36 rocksdb.write.raw.block.micros P50 : 0.750000 P95 : 18.850000 P99 : 19.000000 P100 : 19.000000 COUNT : 18 SUM : 56 rocksdb.l0.slowdown.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.memtable.compaction.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.files.stall.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.hard.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.soft.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.numfiles.in.singlecompaction P50 : 2.000000 P95 : 2.000000 P99 : 2.000000 P100 : 2.000000 COUNT : 2 SUM : 4 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.sst.read.micros P50 : 0.852941 P95 : 12.583333 P99 : 14.516667 P100 : 15.000000 COUNT : 29 SUM : 88 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 116.000000 P95 : 116.000000 P99 : 116.000000 P100 : 116.000000 COUNT : 14761 SUM : 1712276 rocksdb.bytes.per.write P50 : 31.002666 P95 : 48.468637 P99 : 50.509846 P100 : 267.000000 COUNT : 10131 SUM : 355506 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.gc.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0

nodeId: <naming_persistent_service_v2/164.90.68.130:14432> state: STATE_FOLLOWER leaderId: 10.236.191.5:14432 term: 25 conf: ConfigurationEntry [id=LogId [index=339039339, term=25], conf=10.235.191.5:14432,10.235.191.6:14432,10.236.191.5:14432,169.136.78.33:14432,164.90.68.130:14432, oldConf=] targetPriority: -1 electionTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@dc3e64e, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-ElectionTimer-<naming_persistent_service_v2/164.90.68.130:14432>'} voteTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=5000, name='JRaft-VoteTimer-<naming_persistent_service_v2/164.90.68.130:14432>'} stepDownTimer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=false, invoking=false, timeoutMs=2500, name='JRaft-StepDownTimer-<naming_persistent_service_v2/164.90.68.130:14432>'} snapshotTimer: RepeatedTimer{timeout=com.alipay.sofa.jraft.util.timer.DefaultTimer$TimeoutTask$1@16b9834b, stopped=false, running=true, destroyed=false, invoking=false, timeoutMs=1800000, name='JRaft-SnapshotTimer-<naming_persistent_service_v2/164.90.68.130:14432>'} logManager: storage: [339039143, 339039420] diskId: LogId [index=339039420, term=25] appliedId: LogId [index=339039420, term=25] lastSnapshotId: LogId [index=339039339, term=25] fsmCaller: StateMachine [Idle] ballotBox: lastCommittedIndex: 339039420 pendingIndex: 0 pendingMetaQueueSize: 0 snapshotExecutor: lastSnapshotTerm: 25 lastSnapshotIndex: 339039339 term: 25 savingSnapshot: false loadingSnapshot: false stopped: false replicatorGroup: replicators: [] failureReplicators: {} logStorage:

Compaction Stats [default] Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop

L0 1/0 407.71 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0 4 0.111 0 0 L1 1/0 376.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 18.4 5.7 0 2 0.049 1889 1261 Sum 2/0 783.91 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 3.3 3.9 1 6 0.090 1889 1261 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 Uptime(secs): 6074951.6 total, 521.5 interval Flush(GB): cumulative 0.002, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.5 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count

File Read Latency Histogram By Level [default] ** Level 0 read latency histogram (micros): Count: 334 Average: 68.1317 StdDev: 839.36 Min: 0 Median: 0.7731 Max: 13647 Percentiles: P50: 0.77 P75: 1.39 P99: 365.27 P99.9: 12630.60 P99.99: 13647.00

[ 0, 1 ] 216 64.671% 64.671% ############# ( 1, 2 ] 89 26.647% 91.317% ##### ( 2, 3 ] 6 1.796% 93.114% ( 3, 4 ] 3 0.898% 94.012% ( 4, 6 ] 4 1.198% 95.210% ( 6, 10 ] 5 1.497% 96.707% ( 10, 15 ] 1 0.299% 97.006% ( 15, 22 ] 1 0.299% 97.305% ( 22, 34 ] 3 0.898% 98.204% ( 250, 380 ] 3 0.898% 99.102% ( 380, 580 ] 1 0.299% 99.401% ( 6600, 9900 ] 1 0.299% 99.701% ( 9900, 14000 ] 1 0.299% 100.000%

** Level 1 read latency histogram (micros): Count: 131 Average: 125.0382 StdDev: 953.75 Min: 0 Median: 0.9097 Max: 8334 Percentiles: P50: 0.91 P75: 2.42 P99: 7738.50 P99.9: 8334.00 P99.99: 8334.00

[ 0, 1 ] 72 54.962% 54.962% ########### ( 1, 2 ] 25 19.084% 74.046% #### ( 2, 3 ] 3 2.290% 76.336% ( 3, 4 ] 3 2.290% 78.626% ( 4, 6 ] 3 2.290% 80.916% ( 6, 10 ] 12 9.160% 90.076% ## ( 10, 15 ] 6 4.580% 94.656% # ( 15, 22 ] 1 0.763% 95.420% ( 22, 34 ] 1 0.763% 96.183% ( 110, 170 ] 2 1.527% 97.710% ( 170, 250 ] 1 0.763% 98.473% ( 6600, 9900 ] 2 1.527% 100.000%

DB Stats Uptime(secs): 6074951.6 total, 521.5 interval Cumulative writes: 635K writes, 830K keys, 635K commit groups, 1.0 writes per commit group, ingest: 0.68 GB, 0.00 MB/s Cumulative WAL: 635K writes, 628K syncs, 1.01 writes per sync, written: 0.68 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 50 writes, 78 keys, 50 commit groups, 1.0 writes per commit group, ingest: 0.07 MB, 0.00 MB/s Interval WAL: 50 writes, 50 syncs, 0.98 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent

rocksdb.block.cache.miss COUNT : 440 rocksdb.block.cache.hit COUNT : 195 rocksdb.block.cache.add COUNT : 45 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 12 rocksdb.block.cache.index.hit COUNT : 100 rocksdb.block.cache.index.add COUNT : 11 rocksdb.block.cache.index.bytes.insert COUNT : 18000 rocksdb.block.cache.index.bytes.evict COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.filter.bytes.evict COUNT : 0 rocksdb.block.cache.data.miss COUNT : 428 rocksdb.block.cache.data.hit COUNT : 95 rocksdb.block.cache.data.add COUNT : 34 rocksdb.block.cache.data.bytes.insert COUNT : 504976 rocksdb.block.cache.bytes.read COUNT : 513200 rocksdb.block.cache.bytes.write COUNT : 522976 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 0 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 46668872 rocksdb.memtable.miss COUNT : 89 rocksdb.l0.hit COUNT : 1 rocksdb.l1.hit COUNT : 88 rocksdb.l2andup.hit COUNT : 0 rocksdb.compaction.key.drop.new COUNT : 1262 rocksdb.compaction.key.drop.obsolete COUNT : 12464 rocksdb.compaction.key.drop.range_del COUNT : 1261 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 12464 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 830522 rocksdb.number.keys.read COUNT : 46668961 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 726821951 rocksdb.bytes.read COUNT : 41724321231 rocksdb.number.db.seek COUNT : 2 rocksdb.number.db.next COUNT : 1 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 2 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 1062 rocksdb.no.file.closes COUNT : 0 rocksdb.no.file.opens COUNT : 11 rocksdb.no.file.errors COUNT : 0 rocksdb.l0.slowdown.micros COUNT : 0 rocksdb.memtable.compaction.micros COUNT : 0 rocksdb.l0.num.files.stall.micros COUNT : 0 rocksdb.stall.micros COUNT : 0 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.rate.limit.delay.millis COUNT : 0 rocksdb.num.iterators COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.deletes.filtered COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.block.cachecompressed.miss COUNT : 0 rocksdb.block.cachecompressed.hit COUNT : 0 rocksdb.block.cachecompressed.add COUNT : 0 rocksdb.block.cachecompressed.add.failures COUNT : 0 rocksdb.wal.synced COUNT : 628872 rocksdb.wal.bytes COUNT : 726821951 rocksdb.write.self COUNT : 635624 rocksdb.write.other COUNT : 0 rocksdb.write.timeout COUNT : 0 rocksdb.write.wal COUNT : 1271248 rocksdb.compact.read.bytes COUNT : 1681105 rocksdb.compact.write.bytes COUNT : 813016 rocksdb.flush.write.bytes COUNT : 1422505 rocksdb.number.direct.load.table.properties COUNT : 0 rocksdb.number.superversion_acquires COUNT : 217 rocksdb.number.superversion_releases COUNT : 0 rocksdb.number.superversion_cleanups COUNT : 0 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 1 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.overwritten COUNT : 0 rocksdb.blobdb.gc.num.keys.expired COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.overwritten COUNT : 0 rocksdb.blobdb.gc.bytes.expired COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 2 rocksdb.num.iterator.deleted COUNT : 2 rocksdb.db.get.micros P50 : 181.043947 P95 : 558.233978 P99 : 797.840376 P100 : 16500.000000 COUNT : 46668961 SUM : 10247282898 rocksdb.db.write.micros P50 : 18949.058834 P95 : 40646.542304 P99 : 101274.195756 P100 : 4024263.000000 COUNT : 635624 SUM : 13663317656 rocksdb.compaction.times.micros P50 : 41500.000000 P95 : 68870.000000 P99 : 68870.000000 P100 : 68870.000000 COUNT : 3 SUM : 131128 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 23833.333333 P95 : 31185.000000 P99 : 31185.000000 P100 : 31185.000000 COUNT : 5 SUM : 117970 rocksdb.compaction.outfile.sync.micros P50 : 20000.000000 P95 : 31350.000000 P99 : 31669.000000 P100 : 31669.000000 COUNT : 3 SUM : 65779 rocksdb.wal.file.sync.micros P50 : 18977.379753 P95 : 40716.801163 P99 : 102573.042886 P100 : 4024181.000000 COUNT : 628872 SUM : 13607862424 rocksdb.manifest.file.sync.micros P50 : 17881.000000 P95 : 17881.000000 P99 : 17881.000000 P100 : 17881.000000 COUNT : 8 SUM : 133010 rocksdb.table.open.io.micros P50 : 455.000000 P95 : 7394.000000 P99 : 7394.000000 P100 : 7394.000000 COUNT : 11 SUM : 11588 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.get.micros P50 : 7.460076 P95 : 51.000000 P99 : 2500.000000 P100 : 13653.000000 COUNT : 440 SUM : 46487 rocksdb.write.raw.block.micros P50 : 5.607018 P95 : 14.200000 P99 : 64.666667 P100 : 158.000000 COUNT : 536 SUM : 4234 rocksdb.l0.slowdown.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.memtable.compaction.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.files.stall.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.hard.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.soft.rate.limit.delay.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.numfiles.in.singlecompaction P50 : 2.000000 P95 : 2.000000 P99 : 2.000000 P100 : 2.000000 COUNT : 3 SUM : 6 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.sst.read.micros P50 : 0.809365 P95 : 10.571429 P99 : 412.000000 P100 : 13647.000000 COUNT : 484 SUM : 39425 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1035.228087 P95 : 1204.000000 P99 : 1204.000000 P100 : 1204.000000 COUNT : 46668961 SUM : 41724321231 rocksdb.bytes.per.write P50 : 1046.243204 P95 : 2742.205290 P99 : 4016.339240 P100 : 139357.000000 COUNT : 635624 SUM : 726821951 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.gc.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 132072.000000 P95 : 132072.000000 P99 : 132072.000000 P100 : 132072.000000 COUNT : 3 SUM : 389801

zz630 commented 1 year ago

node_metrics

-- <naming_service_metadata/164.90.68.130:14432> 3/22/23 5:40:42 PM =============================================================

-- <naming_service_metadata/164.90.68.130:14432> -- Gauges ---------------------------------------------------------------------- jraft-fsm-caller-disruptor.buffer-size value = 16384 jraft-fsm-caller-disruptor.remaining-capacity value = 16384 jraft-log-manager-disruptor.buffer-size value = 16384 jraft-log-manager-disruptor.remaining-capacity value = 16384 jraft-node-impl-disruptor.buffer-size value = 16384 jraft-node-impl-disruptor.remaining-capacity value = 16384 jraft-read-only-service-disruptor.buffer-size value = 16384 jraft-read-only-service-disruptor.remaining-capacity value = 16384 raft-rpc-client-thread-pool.active value = 0 raft-rpc-client-thread-pool.completed value = 6 raft-rpc-client-thread-pool.pool-size value = 6 raft-rpc-client-thread-pool.queued value = 0 raft-utils-closure-thread-pool.active value = 0 raft-utils-closure-thread-pool.completed value = 4652384 raft-utils-closure-thread-pool.pool-size value = 32 raft-utils-closure-thread-pool.queued value = 0

-- <naming_service_metadata/164.90.68.130:14432> -- Histograms ------------------------------------------------------------------ append-logs-bytes count = 3 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 append-logs-count count = 3 min = 1 max = 1 mean = 1.00 stddev = 0.00 median = 1.00 75% <= 1.00 95% <= 1.00 98% <= 1.00 99% <= 1.00 99.9% <= 1.00 handle-append-entries-count count = 227924 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 replicate-entries-bytes count = 4 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 replicate-entries-count count = 4 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 replicate-inflights-count count = 633 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00

-- <naming_service_metadata/164.90.68.130:14432> -- Timers ---------------------------------------------------------------------- append-logs count = 3 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 55.00 milliseconds max = 55.00 milliseconds mean = 55.00 milliseconds stddev = 0.00 milliseconds median = 55.00 milliseconds 75% <= 55.00 milliseconds 95% <= 55.00 milliseconds 98% <= 55.00 milliseconds 99% <= 55.00 milliseconds 99.9% <= 55.00 milliseconds fsm-commit count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-leader-stop count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-load count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-save count = 3 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-start-following count = 3 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 66.00 milliseconds mean = 32.75 milliseconds stddev = 33.00 milliseconds median = 0.00 milliseconds 75% <= 66.00 milliseconds 95% <= 66.00 milliseconds 98% <= 66.00 milliseconds 99% <= 66.00 milliseconds 99.9% <= 66.00 milliseconds fsm-stop-following count = 2 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds handle-append-entries count = 227925 mean rate = 0.04 calls/second 1-minute rate = 2.00 calls/second 5-minute rate = 1.99 calls/second 15-minute rate = 1.99 calls/second min = 0.00 milliseconds max = 1.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds replicate-entries count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds request-vote count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds save-raft-meta count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 66.00 milliseconds max = 10852.00 milliseconds mean = 5459.00 milliseconds stddev = 5393.00 milliseconds median = 10852.00 milliseconds 75% <= 10852.00 milliseconds 95% <= 10852.00 milliseconds 98% <= 10852.00 milliseconds 99% <= 10852.00 milliseconds 99.9% <= 10852.00 milliseconds truncate-log-prefix count = 3376 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 14.00 milliseconds max = 37.00 milliseconds mean = 17.00 milliseconds stddev = 0.00 milliseconds median = 17.00 milliseconds 75% <= 17.00 milliseconds 95% <= 17.00 milliseconds 98% <= 17.00 milliseconds 99% <= 17.00 milliseconds 99.9% <= 17.00 milliseconds

-- <naming_persistent_service/164.90.68.130:14432> 3/22/23 5:40:43 PM =============================================================

-- <naming_persistent_service/164.90.68.130:14432> -- Gauges ---------------------------------------------------------------------- jraft-fsm-caller-disruptor.buffer-size value = 16384 jraft-fsm-caller-disruptor.remaining-capacity value = 16384 jraft-log-manager-disruptor.buffer-size value = 16384 jraft-log-manager-disruptor.remaining-capacity value = 16384 jraft-node-impl-disruptor.buffer-size value = 16384 jraft-node-impl-disruptor.remaining-capacity value = 16384 jraft-read-only-service-disruptor.buffer-size value = 16384 jraft-read-only-service-disruptor.remaining-capacity value = 16384 raft-rpc-client-thread-pool.active value = 0 raft-rpc-client-thread-pool.completed value = 6 raft-rpc-client-thread-pool.pool-size value = 6 raft-rpc-client-thread-pool.queued value = 0 raft-utils-closure-thread-pool.active value = 0 raft-utils-closure-thread-pool.completed value = 4652384 raft-utils-closure-thread-pool.pool-size value = 32 raft-utils-closure-thread-pool.queued value = 0

-- <naming_persistent_service/164.90.68.130:14432> -- Histograms ------------------------------------------------------------------ append-logs-bytes count = 289337 min = 1184 max = 297317 mean = 41671.93 stddev = 93329.33 median = 13920.00 75% <= 15088.00 95% <= 296401.00 98% <= 297317.00 99% <= 297317.00 99.9% <= 297317.00 append-logs-count count = 289337 min = 1 max = 4 mean = 1.00 stddev = 0.00 median = 1.00 75% <= 1.00 95% <= 1.00 98% <= 1.00 99% <= 1.00 99.9% <= 1.00 fsm-apply-tasks-count count = 294532 min = 1 max = 3 mean = 1.00 stddev = 0.00 median = 1.00 75% <= 1.00 95% <= 1.00 98% <= 1.00 99% <= 1.00 99.9% <= 1.00 handle-append-entries-count count = 234719 min = 0 max = 1 mean = 0.03 stddev = 0.18 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 1.00 99% <= 1.00 99.9% <= 1.00 handle-read-index-entries count = 1239 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 replicate-entries-bytes count = 1158693 min = 1184 max = 275205 mean = 18069.12 stddev = 9179.92 median = 17773.00 75% <= 18845.00 95% <= 34261.00 98% <= 34261.00 99% <= 34261.00 99.9% <= 34261.00 replicate-entries-count count = 1158693 min = 1 max = 2 mean = 1.00 stddev = 0.00 median = 1.00 75% <= 1.00 95% <= 1.00 98% <= 1.00 99% <= 1.00 99.9% <= 1.00 replicate-inflights-count count = 1159865 min = 1 max = 2 mean = 1.00 stddev = 0.00 median = 1.00 75% <= 1.00 95% <= 1.00 98% <= 1.00 99% <= 1.00 99.9% <= 1.00

-- <naming_persistent_service/164.90.68.130:14432> -- Timers ---------------------------------------------------------------------- append-logs count = 289337 mean rate = 0.05 calls/second 1-minute rate = 0.05 calls/second 5-minute rate = 0.05 calls/second 15-minute rate = 0.03 calls/second min = 13.00 milliseconds max = 3532.00 milliseconds mean = 41.84 milliseconds stddev = 48.86 milliseconds median = 21.00 milliseconds 75% <= 39.00 milliseconds 95% <= 174.00 milliseconds 98% <= 174.00 milliseconds 99% <= 174.00 milliseconds 99.9% <= 174.00 milliseconds fsm-apply-tasks count = 294532 mean rate = 0.05 calls/second 1-minute rate = 0.05 calls/second 5-minute rate = 0.05 calls/second 15-minute rate = 0.03 calls/second min = 0.00 milliseconds max = 1544.00 milliseconds mean = 0.64 milliseconds stddev = 1.19 milliseconds median = 0.00 milliseconds 75% <= 1.00 milliseconds 95% <= 4.00 milliseconds 98% <= 4.00 milliseconds 99% <= 4.00 milliseconds 99.9% <= 4.00 milliseconds fsm-commit count = 294533 mean rate = 0.05 calls/second 1-minute rate = 0.05 calls/second 5-minute rate = 0.05 calls/second 15-minute rate = 0.03 calls/second min = 0.00 milliseconds max = 1544.00 milliseconds mean = 0.70 milliseconds stddev = 1.33 milliseconds median = 0.00 milliseconds 75% <= 1.00 milliseconds 95% <= 4.00 milliseconds 98% <= 4.00 milliseconds 99% <= 5.00 milliseconds 99.9% <= 5.00 milliseconds fsm-leader-stop count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-load count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-save count = 2883 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 1.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-start-following count = 3 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 78.00 milliseconds mean = 38.71 milliseconds stddev = 39.00 milliseconds median = 0.00 milliseconds 75% <= 78.00 milliseconds 95% <= 78.00 milliseconds 98% <= 78.00 milliseconds 99% <= 78.00 milliseconds 99.9% <= 78.00 milliseconds fsm-stop-following count = 2 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds handle-append-entries count = 234719 mean rate = 0.04 calls/second 1-minute rate = 2.04 calls/second 5-minute rate = 2.04 calls/second 15-minute rate = 2.03 calls/second min = 0.00 milliseconds max = 1.00 milliseconds mean = 0.10 milliseconds stddev = 0.30 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 1.00 milliseconds 98% <= 1.00 milliseconds 99% <= 1.00 milliseconds 99.9% <= 1.00 milliseconds handle-read-index count = 1239 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds read-index count = 208 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds replicate-entries count = 1158693 mean rate = 0.19 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 1.00 milliseconds max = 278.00 milliseconds mean = 12.15 milliseconds stddev = 24.73 milliseconds median = 3.00 milliseconds 75% <= 20.00 milliseconds 95% <= 29.00 milliseconds 98% <= 126.00 milliseconds 99% <= 126.00 milliseconds 99.9% <= 126.00 milliseconds request-vote count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds save-raft-meta count = 3 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 78.00 milliseconds max = 10853.00 milliseconds mean = 5465.50 milliseconds stddev = 5387.50 milliseconds median = 10853.00 milliseconds 75% <= 10853.00 milliseconds 95% <= 10853.00 milliseconds 98% <= 10853.00 milliseconds 99% <= 10853.00 milliseconds 99.9% <= 10853.00 milliseconds truncate-log-prefix count = 3375 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 16.00 milliseconds max = 23.00 milliseconds mean = 17.00 milliseconds stddev = 0.00 milliseconds median = 17.00 milliseconds 75% <= 17.00 milliseconds 95% <= 17.00 milliseconds 98% <= 17.00 milliseconds 99% <= 17.00 milliseconds 99.9% <= 17.00 milliseconds

-- <naming_instance_metadata/164.90.68.130:14432> 3/22/23 5:40:44 PM =============================================================

-- <naming_instance_metadata/164.90.68.130:14432> -- Gauges ---------------------------------------------------------------------- jraft-fsm-caller-disruptor.buffer-size value = 16384 jraft-fsm-caller-disruptor.remaining-capacity value = 16384 jraft-log-manager-disruptor.buffer-size value = 16384 jraft-log-manager-disruptor.remaining-capacity value = 16384 jraft-node-impl-disruptor.buffer-size value = 16384 jraft-node-impl-disruptor.remaining-capacity value = 16384 jraft-read-only-service-disruptor.buffer-size value = 16384 jraft-read-only-service-disruptor.remaining-capacity value = 16384 raft-rpc-client-thread-pool.active value = 0 raft-rpc-client-thread-pool.completed value = 10 raft-rpc-client-thread-pool.pool-size value = 10 raft-rpc-client-thread-pool.queued value = 0 raft-utils-closure-thread-pool.active value = 0 raft-utils-closure-thread-pool.completed value = 4652384 raft-utils-closure-thread-pool.pool-size value = 32 raft-utils-closure-thread-pool.queued value = 0

-- <naming_instance_metadata/164.90.68.130:14432> -- Histograms ------------------------------------------------------------------ append-logs-bytes count = 4 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 append-logs-count count = 4 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 handle-append-entries-count count = 3844885 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 replicate-entries-bytes count = 4 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 replicate-entries-count count = 4 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00 replicate-inflights-count count = 634 min = 0 max = 0 mean = 0.00 stddev = 0.00 median = 0.00 75% <= 0.00 95% <= 0.00 98% <= 0.00 99% <= 0.00 99.9% <= 0.00

-- <naming_instance_metadata/164.90.68.130:14432> -- Timers ---------------------------------------------------------------------- append-logs count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-commit count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-leader-stop count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-load count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-save count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-start-following count = 5 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-stop-following count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 907.00 milliseconds max = 907.00 milliseconds mean = 907.00 milliseconds stddev = 0.00 milliseconds median = 907.00 milliseconds 75% <= 907.00 milliseconds 95% <= 907.00 milliseconds 98% <= 907.00 milliseconds 99% <= 907.00 milliseconds 99.9% <= 907.00 milliseconds handle-append-entries count = 3844886 mean rate = 0.63 calls/second 1-minute rate = 2.00 calls/second 5-minute rate = 1.99 calls/second 15-minute rate = 1.99 calls/second min = 0.00 milliseconds max = 1.00 milliseconds mean = 0.00 milliseconds stddev = 0.05 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 1.00 milliseconds pre-vote count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 2.00 milliseconds max = 11.00 milliseconds mean = 6.25 milliseconds stddev = 3.83 milliseconds median = 9.00 milliseconds 75% <= 11.00 milliseconds 95% <= 11.00 milliseconds 98% <= 11.00 milliseconds 99% <= 11.00 milliseconds 99.9% <= 11.00 milliseconds replicate-entries count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds request-vote count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds save-raft-meta count = 7 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds truncate-log-prefix count = 3375 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 15.00 milliseconds max = 37.00 milliseconds mean = 18.00 milliseconds stddev = 0.00 milliseconds median = 18.00 milliseconds 75% <= 18.00 milliseconds 95% <= 18.00 milliseconds 98% <= 18.00 milliseconds 99% <= 18.00 milliseconds 99.9% <= 18.00 milliseconds

-- <naming_persistent_service_v2/164.90.68.130:14432> 3/22/23 5:40:45 PM =============================================================

-- <naming_persistent_service_v2/164.90.68.130:14432> -- Gauges ---------------------------------------------------------------------- jraft-fsm-caller-disruptor.buffer-size value = 16384 jraft-fsm-caller-disruptor.remaining-capacity value = 16384 jraft-log-manager-disruptor.buffer-size value = 16384 jraft-log-manager-disruptor.remaining-capacity value = 16384 jraft-node-impl-disruptor.buffer-size value = 16384 jraft-node-impl-disruptor.remaining-capacity value = 16384 jraft-read-only-service-disruptor.buffer-size value = 16384 jraft-read-only-service-disruptor.remaining-capacity value = 16384 raft-rpc-client-thread-pool.active value = 0 raft-rpc-client-thread-pool.completed value = 6 raft-rpc-client-thread-pool.pool-size value = 6 raft-rpc-client-thread-pool.queued value = 0 raft-utils-closure-thread-pool.active value = 0 raft-utils-closure-thread-pool.completed value = 4652384 raft-utils-closure-thread-pool.pool-size value = 32 raft-utils-closure-thread-pool.queued value = 0

-- <naming_persistent_service_v2/164.90.68.130:14432> -- Histograms ------------------------------------------------------------------ append-logs-bytes count = 625497 min = 603 max = 11152 mean = 1193.61 stddev = 600.21 median = 1033.00 75% <= 1106.00 95% <= 2212.00 98% <= 2212.00 99% <= 4106.00 99.9% <= 5185.00 append-logs-count count = 625497 min = 1 max = 17 mean = 1.22 stddev = 0.65 median = 1.00 75% <= 1.00 95% <= 2.00 98% <= 2.00 99% <= 4.00 99.9% <= 8.00 fsm-apply-tasks-count count = 767672 min = 1 max = 16 mean = 1.25 stddev = 0.52 median = 1.00 75% <= 1.00 95% <= 2.00 98% <= 2.00 99% <= 3.00 99.9% <= 5.00 handle-append-entries-count count = 239176 min = 0 max = 2 mean = 0.07 stddev = 0.29 median = 0.00 75% <= 0.00 95% <= 1.00 98% <= 1.00 99% <= 1.00 99.9% <= 2.00 replicate-entries-bytes count = 3038397 min = 602 max = 2931 mean = 937.19 stddev = 349.65 median = 1002.00 75% <= 1027.00 95% <= 2058.00 98% <= 2058.00 99% <= 2058.00 99.9% <= 2058.00 replicate-entries-count count = 3038397 min = 1 max = 3 mean = 1.13 stddev = 0.34 median = 1.00 75% <= 1.00 95% <= 2.00 98% <= 2.00 99% <= 2.00 99.9% <= 2.00 replicate-inflights-count count = 3039971 min = 1 max = 4 mean = 1.35 stddev = 0.58 median = 1.00 75% <= 2.00 95% <= 3.00 98% <= 3.00 99% <= 3.00 99.9% <= 3.00

-- <naming_persistent_service_v2/164.90.68.130:14432> -- Timers ---------------------------------------------------------------------- append-logs count = 625497 mean rate = 0.10 calls/second 1-minute rate = 0.13 calls/second 5-minute rate = 0.10 calls/second 15-minute rate = 0.07 calls/second min = 11.00 milliseconds max = 4024.00 milliseconds mean = 19.25 milliseconds stddev = 2.35 milliseconds median = 19.00 milliseconds 75% <= 21.00 milliseconds 95% <= 23.00 milliseconds 98% <= 23.00 milliseconds 99% <= 23.00 milliseconds 99.9% <= 23.00 milliseconds fsm-apply-tasks count = 767672 mean rate = 0.13 calls/second 1-minute rate = 0.13 calls/second 5-minute rate = 0.10 calls/second 15-minute rate = 0.07 calls/second min = 0.00 milliseconds max = 1.00 milliseconds mean = 0.20 milliseconds stddev = 0.40 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 1.00 milliseconds 98% <= 1.00 milliseconds 99% <= 1.00 milliseconds 99.9% <= 1.00 milliseconds fsm-commit count = 767673 mean rate = 0.13 calls/second 1-minute rate = 0.13 calls/second 5-minute rate = 0.10 calls/second 15-minute rate = 0.07 calls/second min = 0.00 milliseconds max = 2.00 milliseconds mean = 0.21 milliseconds stddev = 0.41 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 1.00 milliseconds 98% <= 1.00 milliseconds 99% <= 1.00 milliseconds 99.9% <= 2.00 milliseconds fsm-leader-stop count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-load count = 1 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-snapshot-save count = 3344 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 1.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds fsm-start-following count = 3 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 17.00 milliseconds mean = 8.44 milliseconds stddev = 8.50 milliseconds median = 0.00 milliseconds 75% <= 17.00 milliseconds 95% <= 17.00 milliseconds 98% <= 17.00 milliseconds 99% <= 17.00 milliseconds 99.9% <= 17.00 milliseconds fsm-stop-following count = 2 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds handle-append-entries count = 239177 mean rate = 0.04 calls/second 1-minute rate = 2.13 calls/second 5-minute rate = 2.12 calls/second 15-minute rate = 2.09 calls/second min = 0.00 milliseconds max = 23.00 milliseconds mean = 0.77 milliseconds stddev = 1.07 milliseconds median = 1.00 milliseconds 75% <= 1.00 milliseconds 95% <= 1.00 milliseconds 98% <= 1.00 milliseconds 99% <= 2.00 milliseconds 99.9% <= 12.00 milliseconds replicate-entries count = 3038397 mean rate = 0.51 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 670.00 milliseconds mean = 29.25 milliseconds stddev = 100.36 milliseconds median = 3.00 milliseconds 75% <= 19.00 milliseconds 95% <= 102.00 milliseconds 98% <= 401.00 milliseconds 99% <= 670.00 milliseconds 99.9% <= 670.00 milliseconds request-vote count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds save-raft-meta count = 3 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 61.00 milliseconds max = 10853.00 milliseconds mean = 5457.00 milliseconds stddev = 5396.00 milliseconds median = 10853.00 milliseconds 75% <= 10853.00 milliseconds 95% <= 10853.00 milliseconds 98% <= 10853.00 milliseconds 99% <= 10853.00 milliseconds 99.9% <= 10853.00 milliseconds truncate-log-prefix count = 3375 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 16.00 milliseconds max = 37.00 milliseconds mean = 17.00 milliseconds stddev = 0.00 milliseconds median = 17.00 milliseconds 75% <= 17.00 milliseconds 95% <= 17.00 milliseconds 98% <= 17.00 milliseconds 99% <= 17.00 milliseconds 99.9% <= 17.00 milliseconds

zz630 commented 1 year ago

thread_pool_metrics

3/22/23 5:40:45 PM =============================================================

-- Timers ---------------------------------------------------------------------- scheduledThreadPool.JRaft-Global-ElectionTimer count = 417879 mean rate = 0.07 calls/second 1-minute rate = 1.28 calls/second 5-minute rate = 1.28 calls/second 15-minute rate = 1.27 calls/second min = 0.00 milliseconds max = 0.39 milliseconds mean = 0.01 milliseconds stddev = 0.01 milliseconds median = 0.01 milliseconds 75% <= 0.01 milliseconds 95% <= 0.02 milliseconds 98% <= 0.02 milliseconds 99% <= 0.02 milliseconds 99.9% <= 0.02 milliseconds scheduledThreadPool.JRaft-Global-SnapshotTimer count = 13499 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.01 milliseconds max = 0.03 milliseconds mean = 0.02 milliseconds stddev = 0.00 milliseconds median = 0.02 milliseconds 75% <= 0.02 milliseconds 95% <= 0.02 milliseconds 98% <= 0.02 milliseconds 99% <= 0.02 milliseconds 99.9% <= 0.02 milliseconds scheduledThreadPool.JRaft-Global-StepDownTimer count = 8811048 mean rate = 1.48 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 910.03 milliseconds mean = 38.29 milliseconds stddev = 182.65 milliseconds median = 0.01 milliseconds 75% <= 0.02 milliseconds 95% <= 0.05 milliseconds 98% <= 910.03 milliseconds 99% <= 910.03 milliseconds 99.9% <= 910.03 milliseconds scheduledThreadPool.JRaft-Global-VoteTimer count = 4 mean rate = 0.00 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.00 milliseconds mean = 0.00 milliseconds stddev = 0.00 milliseconds median = 0.00 milliseconds 75% <= 0.00 milliseconds 95% <= 0.00 milliseconds 98% <= 0.00 milliseconds 99% <= 0.00 milliseconds 99.9% <= 0.00 milliseconds scheduledThreadPool.JRaft-Node-ScheduleThreadPool count = 175700964 mean rate = 29.43 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.00 milliseconds max = 0.02 milliseconds mean = 0.01 milliseconds stddev = 0.00 milliseconds median = 0.01 milliseconds 75% <= 0.01 milliseconds 95% <= 0.01 milliseconds 98% <= 0.01 milliseconds 99% <= 0.01 milliseconds 99.9% <= 0.02 milliseconds threadPool.JRAFT_CLOSURE_EXECUTOR count = 4652384 mean rate = 0.77 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.01 calls/second 15-minute rate = 0.01 calls/second min = 0.03 milliseconds max = 109.95 milliseconds mean = 9.89 milliseconds stddev = 13.81 milliseconds median = 0.13 milliseconds 75% <= 29.39 milliseconds 95% <= 29.39 milliseconds 98% <= 29.39 milliseconds 99% <= 29.39 milliseconds 99.9% <= 29.39 milliseconds threadPool.JRAFT_RPC_CLOSURE_EXECUTOR count = 175700976 mean rate = 29.43 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.02 milliseconds max = 2.23 milliseconds mean = 0.30 milliseconds stddev = 0.35 milliseconds median = 0.10 milliseconds 75% <= 0.63 milliseconds 95% <= 0.83 milliseconds 98% <= 0.96 milliseconds 99% <= 1.36 milliseconds 99.9% <= 2.23 milliseconds threadPool.JRaft-RPC-Processor count = 7402486 mean rate = 1.22 calls/second 1-minute rate = 1.23 calls/second 5-minute rate = 1.22 calls/second 15-minute rate = 1.22 calls/second min = 0.00 milliseconds max = 0.02 milliseconds mean = 0.01 milliseconds stddev = 0.00 milliseconds median = 0.01 milliseconds 75% <= 0.01 milliseconds 95% <= 0.01 milliseconds 98% <= 0.01 milliseconds 99% <= 0.01 milliseconds 99.9% <= 0.02 milliseconds threadPool.grpc-default-executor count = 748497 mean rate = 0.13 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 0.01 milliseconds max = 11361.62 milliseconds mean = 3663.06 milliseconds stddev = 4984.57 milliseconds median = 0.02 milliseconds 75% <= 9340.91 milliseconds 95% <= 11361.62 milliseconds 98% <= 11361.62 milliseconds 99% <= 11361.62 milliseconds 99.9% <= 11361.62 milliseconds

fengjiachun commented 1 year ago

你的业务状态机处理的够快吗？业务状态执行慢会导致 jraft 里面的各种 buf 堆积， 如果你业务状态机不够快，同时希望内存占用小一些，那么就调小 disruptor 的 buf size，每个 disruptor 的 buf size 都在某个 xxxOption 里，可以调小试试

zz630 commented 1 year ago

业务状态机没研究过，就是nacos的NacosStateMachine。实例上下线不频繁，感觉没延迟。 磁盘io await在几毫秒到十几毫秒浮动

fengjiachun commented 1 year ago

如果觉得不慢，调小 disruptor 的 buf size

fengjiachun commented 1 year ago

如果还未解决可再打开 issue 继续讨论