dlg99
commented
2 years ago Do you have rereplicationEntryBatchSize set to a large value? try setting it back to default (10).
Open gaozhangmin opened 2 years ago
dlg99
commented
2 years ago Do you have rereplicationEntryBatchSize set to a large value? try setting it back to default (10).
gaozhangmin
commented
2 years ago @dlg99 it's 500, it's because of this setting?
StevenLuMT
commented
2 years ago can you provide a bookie configuration? I help you analyze it @gaozhangmin
horizonzy
commented
2 years ago After research for a long time. We found that is bookie problem, the request from ReplicationWorker is too many.
The shutdown bookie holds many ledgers, when it shutdown, the Auditor mark many ledgers to underreplication.
And there are many ReplicationWorker to replicate ledger, the config rereplicationEntryBatchSize is 500, so every ReplicationWorker will send 500 read request to bookie servers, so the bookie server receives lots of reuqest, it will allocate direct memory for reuqest.
The release operation is not catching up allocate operation, so the PoolChunk is more and more until it reach maxDirectMemory.
@gaozhangmin supply two heap dumps file, the less is dumpped when replicate operation start, The more file is dumpped when the replicate for a while.
I found that PoolChunk is 244 in more, 120 in less. The PoolChunk direct memory is 4M in bookie, so it increase 124 * 4M direct memory than less.
And there is another issue we found, if user config DbLedgerStorage, when it start, it will occupy 1/2 direct memory for readCache and writeCache, it's unpooled but cuupy direct memory.
In the Direct memory pool, it only has 1/2 direct memory to allocate, it will cause oom easier.
horizonzy
commented
2 years ago And we found the io.netty.buffer.PooledByteBufAllocator#directArenas array size is 80, it also cause oom easier.
horizonzy
commented
2 years ago So this is a normal case, you can reduce rereplicationEntryBatchSize value to decrease bookie server reque frequency.
If no more confusion, can be close.
yapxue
commented
2 years ago @horizonzy I also suffered OutOfDirectMemoryError when doing performance test. I send very huge writes and bookie will OOM and restart all the time. But in my case, replica recovery is disabled. If it is caused by allocation fatser than release, why the direct memory is still not released after a long time even if I stopped prodcuer and entries rate is 0. Here is a snapshot of direct memory usage.
horizonzy
commented
2 years ago BytBuf.release maybe not free PoolChunk. If The ByteBuf is cached by PoolThreadCache. The direct memory can't be release.
StevenLuMT
commented
2 years ago @horizonzy I also suffered OutOfDirectMemoryError when doing performance test. I send very huge writes and bookie will OOM and restart all the time. But in my case, replica recovery is disabled. If it is caused by allocation fatser than release, why the direct memory is still not released after a long time even if I stopped prodcuer and entries rate is 0. Here is a snapshot of direct memory usage.
Your scene is different,I will discuss the details with you offline @yapxue
jimmycxm
commented
2 years ago @horizonzy I also suffered OutOfDirectMemoryError when doing performance test. I send very huge writes and bookie will OOM and restart all the time. But in my case, replica recovery is disabled. If it is caused by allocation fatser than release, why the direct memory is still not released after a long time even if I stopped prodcuer and entries rate is 0. Here is a snapshot of direct memory usage.
Got same issue. Is the problem resolved?
Bookkeeper Version: 4.14.4
Our prod environment went wrong last week, all bookies were killed because of direct memory OOM, this happened after one bookie's disk was broken, we tried to offline this bookie. After auditBookie triggered, all the bookies Direct Memory keep increase, it seem that, there is memory leak problem.
The ReplicateWorker log: x.x.x.x is the ip of lost bookie
And there are bookies quarantined by brokers continuous, all bookies are crashed at last.
Note: autoRecoveryDaemonEnabled=true, we run autoRecovery service inside the bookie service. After the bookie was lost because of disk failure, we also run a decommission command to offline this bookie. (I thinks it s a duplicate trigger of auditing bookie)
NUM_ENTRIES_UNABLE_TO_READ_FOR_REPLICATION metrics