search

AutoMQ / automq

AutoMQ is a cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency.
https://www.automq.com/docs
Other
3.79k stars 216 forks source link

OOM while compacting objects #341

Closed Chillax-0v0 closed 1 year ago

Chillax-0v0 commented 1 year ago

server.log:

[2023-10-13 14:28:02,689] INFO [Stream id=0 epoch0] stream objects compaction finished, no compaction happened (com.automq.stream.s3.S3Stream)
[2023-10-13 14:28:02,689] INFO [Stream id=1 epoch0] stream objects compaction finished, no compaction happened (com.automq.stream.s3.S3Stream)
...
[2023-10-13 14:28:12,736] INFO try trim WAL to 421653442560 (com.automq.stream.s3.S3Storage)
[2023-10-13 14:28:20,857] INFO log append cost, permitAcquireFail={count=0, max=0.0, mean=0.0}, remainingPermit=1073729131/1073741824, append={count=71717, max=1991135.0, mean=18643.603413416622}, callback={count=71719, max=4.9201698E7, mean=325577.6692
647695}, ack={count=56761, max=2.8206865E7, mean=79708.46827927626} (kafka.log.streamaspect.ElasticLog)
[2023-10-13 14:28:26,264] INFO [DefaultObjectWriter objId=13974] upload s3 metrics: total_parts: 1, upload_part_time_avg: 524.0, ready_close_time 187, total_upload_time 1788, total_upload_size 10576843 (com.automq.stream.s3.operator.DefaultS3Operator)
[2023-10-13 14:28:26,449] INFO [Stream id=158 epoch0] stream objects compaction finished, compaction summary: CompactionSummary{streamId=158, startOffset=0, endOffset=1714648, timeCostInMs=23716, totalObjectSize=7060862338, sourceObjectsCount=174, targe
tObjectCount=2, smallSizeCopyWriteCount=0} (com.automq.stream.s3.S3Stream)
[2023-10-13 14:28:26,449] INFO [Stream id=159 epoch0] stream objects compaction finished, no compaction happened (com.automq.stream.s3.S3Stream)
[2023-10-13 14:28:26,472] INFO [StreamObjectsCompactionTask streamId=160]  prepared 2 groups for compaction (com.automq.stream.s3.StreamObjectsCompactionTask)
[2023-10-13 14:28:27,338] WARN log cache size 1073941746 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:28:27,338] WARN log cache size 1073941746 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
...
[2023-10-13 14:28:35,044] WARN log cache size 1073941746 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:28:35,061] INFO try trim WAL to 422586331136 (com.automq.stream.s3.S3Storage)
[2023-10-13 14:28:35,265] INFO log append cost, permitAcquireFail={count=0, max=0.0, mean=0.0}, remainingPermit=1073741373/1073741824, append={count=33485, max=7.806430324E9, mean=1188347.0346722412}, callback={count=33486, max=7.927332538E9, mean=19487
54.117631249}, ack={count=25130, max=3.7752323E7, mean=170326.39426979705} (kafka.log.streamaspect.ElasticLog)
[2023-10-13 14:28:35,266] INFO [KafkaApi-0] produce cost, produce={count=7074, max=6.8112862E7, mean=1636818.2804636697} callback={count=7074, max=7.926791195E9, mean=2739722.592875318} ack={count=7074, max=1.264117E7, mean=15292.45603618886} (kafka.ser
ver.KafkaApis)
[2023-10-13 14:28:36,705] INFO [StreamObjectCopier objId=13986] upload s3 metrics: total_parts: 81, upload_part_time_avg: 548.4444444444445, ready_close_time 8580, total_upload_time 10229, total_upload_size 7732 (com.automq.stream.s3.operator.DefaultS3O
perator)
[2023-10-13 14:28:45,266] INFO log append cost, permitAcquireFail={count=0, max=0.0, mean=0.0}, remainingPermit=1073585267/1073741824, append={count=58717, max=4.2002993E7, mean=23040.46107600865}, callback={count=58714, max=1.34471905E8, mean=401224.99
46350104}, ack={count=46526, max=2.094663E7, mean=81466.06473799596} (kafka.log.streamaspect.ElasticLog)
[2023-10-13 14:28:45,267] INFO [KafkaApi-0] produce cost, produce={count=16855, max=7.942261408E9, mean=3336015.429071492} callback={count=16854, max=7.942232825E9, mean=2853125.652426724} ack={count=16855, max=1968778.0, mean=12161.82029071492} (kafka.
server.KafkaApis)
[2023-10-13 14:28:46,255] WARN log cache size 1073774901 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:28:46,256] WARN log cache size 1073774913 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
...
...
...
[2023-10-13 14:31:11,225] WARN log cache size 1074010646 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:31:11,243] WARN log cache size 1074010646 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:31:11,295] ERROR prepare and write wal object failed (com.automq.stream.s3.compact.CompactionUploader)
java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: Cannot reserve 32847540 bytes of direct buffer memory (allocated: 4286579045, limit: 4294967296)
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:722)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Cannot reserve 32847540 bytes of direct buffer memory (allocated: 4286579045, limit: 4294967296)
        at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
        at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121)
        at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:649)
        at io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:638)
        at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:214)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        at io.netty.buffer.CompositeByteBuf.allocBuffer(CompositeByteBuf.java:1879)
        at io.netty.buffer.CompositeByteBuf.consolidate0(CompositeByteBuf.java:1758)
        at io.netty.buffer.CompositeByteBuf.consolidateIfNeeded(CompositeByteBuf.java:571)
        at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:266)
        at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:222)
        at com.automq.stream.s3.compact.operator.DataBlockWriter.uploadWaitingList(DataBlockWriter.java:86)
        at com.automq.stream.s3.compact.operator.DataBlockWriter.write(DataBlockWriter.java:65)
        at com.automq.stream.s3.compact.CompactionUploader.lambda$prepareObjectAndWrite$5(CompactionUploader.java:94)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
        ... 7 more
[2023-10-13 14:31:11,324] WARN log cache size 1074010646 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:31:11,323] WARN log cache size 1074010646 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:31:11,314] WARN log cache size 1074010646 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:31:11,351] ERROR [CompactionManager id=0] Error while uploading compaction objects (com.automq.stream.s3.compact.CompactionManager)
java.util.concurrent.CompletionException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:722)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:726)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
        at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
        at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:147)
        at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
        at io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:307)
        at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:265)
        at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:222)
        at com.automq.stream.s3.compact.operator.DataBlockWriter.uploadWaitingList(DataBlockWriter.java:86)
        at com.automq.stream.s3.compact.operator.DataBlockWriter.forceUpload(DataBlockWriter.java:79)
        at com.automq.stream.s3.compact.CompactionUploader.forceUploadWAL(CompactionUploader.java:140)
        at com.automq.stream.s3.compact.CompactionManager.lambda$compactWALObjects$30(CompactionManager.java:421)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
        ... 10 more
[2023-10-13 14:31:11,351] ERROR [CompactionManager id=0] Error while compacting objects  (com.automq.stream.s3.compact.CompactionManager)
java.lang.IllegalArgumentException: Error while uploading compaction objects
        at com.automq.stream.s3.compact.CompactionManager.compactWALObjects(CompactionManager.java:427)
        at com.automq.stream.s3.compact.CompactionManager.buildCompactRequest(CompactionManager.java:278)
        at com.automq.stream.s3.compact.CompactionManager.lambda$compact$4(CompactionManager.java:134)
        at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
        at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309)
        at com.automq.stream.s3.compact.CompactionManager.compact(CompactionManager.java:130)
        at com.automq.stream.s3.compact.CompactionManager.lambda$start$2(CompactionManager.java:110)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.CompletionException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:722)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:726)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        ... 1 more
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
        at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
        at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:147)
        at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
        at io.netty.buffer.CompositeByteBuf.addComponent0(CompositeByteBuf.java:307)
        at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:265)
        at io.netty.buffer.CompositeByteBuf.addComponent(CompositeByteBuf.java:222)
        at com.automq.stream.s3.compact.operator.DataBlockWriter.uploadWaitingList(DataBlockWriter.java:86)
        at com.automq.stream.s3.compact.operator.DataBlockWriter.forceUpload(DataBlockWriter.java:79)
        at com.automq.stream.s3.compact.CompactionUploader.forceUploadWAL(CompactionUploader.java:140)
        at com.automq.stream.s3.compact.CompactionManager.lambda$compactWALObjects$30(CompactionManager.java:421)
        at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
        ... 10 more
[2023-10-13 14:31:11,352] ERROR [CompactionManager id=0] Sanity check failed, stream 165 is missing after compact (com.automq.stream.s3.compact.CompactionManager)
[2023-10-13 14:31:11,352] ERROR [CompactionManager id=0] Sanity check failed, compaction result is illegal (com.automq.stream.s3.compact.CompactionManager)
[2023-10-13 14:31:11,352] INFO [CompactionManager id=0] Compaction complete, total cost 19688 ms, result FAILED (com.automq.stream.s3.compact.CompactionManager)
[2023-10-13 14:31:11,450] WARN log cache size 1074010646 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
[2023-10-13 14:31:11,450] WARN log cache size 1074010646 is larger than 1073741824, wait 100ms (com.automq.stream.s3.S3Storage)
Chillax-0v0 commented 1 year ago

controller.log:

[2023-10-13 14:28:02,421] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 13915 success, compacted objects: [], stream objects: [StreamObject(objectId=13917, objectSize=48466960, streamId=171, startOffset=4322666, endOffset=4334340), StreamObject(objectId=13920, objectSize=47454187, streamId=206, startOffset=4362350, endOffset=4373785), StreamObject(objectId=13916, objectSize=46996996, streamId=166, startOffset=4391602, endOffset=4402924), StreamObject(objectId=13918, objectSize=50018397, streamId=182, startOffset=4325521, endOffset=4337566), StreamObject(objectId=13919, objectSize=50150288, streamId=191, startOffset=4355645, endOffset=4367717), StreamObject(objectId=13923, objectSize=47789855, streamId=224, startOffset=4344226, endOffset=4355737), StreamObject(objectId=13921, objectSize=53700998, streamId=218, startOffset=4337891, endOffset=4350827), StreamObject(objectId=13925, objectSize=50200489, streamId=228, startOffset=4333237, endOffset=4345324), StreamObject(objectId=13924, objectSize=49854008, streamId=226, startOffset=4283036, endOffset=4295038), StreamObject(objectId=13922, objectSize=48292324, streamId=221, startOffset=4264677, endOffset=4276305), StreamObject(objectId=13926, objectSize=45888174, streamId=231, startOffset=4311128, endOffset=4322174)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:07,935] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 13939 success, compacted objects: [], stream objects: [StreamObject(objectId=13942, objectSize=54320532, streamId=209, startOffset=4338953, endOffset=4352017), StreamObject(objectId=13943, objectSize=50316804, streamId=212, startOffset=4316599, endOffset=4328703), StreamObject(objectId=13946, objectSize=57390756, streamId=222, startOffset=4270961, endOffset=4284765), StreamObject(objectId=13940, objectSize=52599377, streamId=177, startOffset=4345565, endOffset=4358208), StreamObject(objectId=13941, objectSize=50096793, streamId=183, startOffset=4286310, endOffset=4298355), StreamObject(objectId=13947, objectSize=56154056, streamId=225, startOffset=4258868, endOffset=4272376), StreamObject(objectId=13949, objectSize=55998946, streamId=230, startOffset=4274236, endOffset=4287714), StreamObject(objectId=13944, objectSize=55537268, streamId=219, startOffset=4272704, endOffset=4286050), StreamObject(objectId=13945, objectSize=56419500, streamId=220, startOffset=4313867, endOffset=4327433), StreamObject(objectId=13948, objectSize=50207443, streamId=227, startOffset=4254703, endOffset=4266784)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:12,729] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 13914 success, compacted objects: [], stream objects: [StreamObject(objectId=13927, objectSize=47643541, streamId=158, startOffset=4811607, endOffset=4823156), StreamObject(objectId=13928, objectSize=51082286, streamId=160, startOffset=4843158, endOffset=4855538), StreamObject(objectId=13929, objectSize=45196272, streamId=162, startOffset=4823246, endOffset=4834200), StreamObject(objectId=13930, objectSize=47228936, streamId=174, startOffset=4842895, endOffset=4854343), StreamObject(objectId=13931, objectSize=47034842, streamId=192, startOffset=4842387, endOffset=4853789), StreamObject(objectId=13932, objectSize=47967148, streamId=195, startOffset=4819221, endOffset=4830853), StreamObject(objectId=13935, objectSize=46556093, streamId=223, startOffset=4818132, endOffset=4829419), StreamObject(objectId=13933, objectSize=51628188, streamId=211, startOffset=4828962, endOffset=4841474), StreamObject(objectId=13936, objectSize=52876704, streamId=229, startOffset=4774987, endOffset=4787805), StreamObject(objectId=13934, objectSize=47694086, streamId=217, startOffset=4745212, endOffset=4756774), StreamObject(objectId=13937, objectSize=43798161, streamId=232, startOffset=4818247, endOffset=4828866)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:13,289] INFO [Controller 0] [CommitStreamObject]: stream object: 13938 commit success, compacted objects: [1, 52, 69, 103, 137, 171, 205, 239, 273, 290, 324, 358, 392, 426, 460, 494, 528, 562, 596, 630, 664, 681, 715, 749, 783, 817, 834, 868, 902, 936, 970, 1004, 2869, 1056, 1093, 1121, 1157, 1192, 1227, 1262, 1297, 1332, 1367, 1384, 1418, 1452, 1486, 1520, 1554, 1589, 1623, 1657, 1691, 1725, 1742, 1776, 1810, 1844, 1878, 1912, 1946, 1980, 2002, 2031, 2065, 2099, 2133, 2167, 2201, 2235, 2269, 2303, 2337, 2371, 2388, 2422, 2456, 2490, 2524, 2541] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:18,163] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 13951 success, compacted objects: [], stream objects: [StreamObject(objectId=13952, objectSize=44428551, streamId=166, startOffset=4402924, endOffset=4413621), StreamObject(objectId=13957, objectSize=48642595, streamId=218, startOffset=4350827, endOffset=4362536), StreamObject(objectId=13953, objectSize=49793246, streamId=171, startOffset=4334340, endOffset=4346322), StreamObject(objectId=13954, objectSize=48836623, streamId=182, startOffset=4337566, endOffset=4349311), StreamObject(objectId=13960, objectSize=48678662, streamId=226, startOffset=4295038, endOffset=4306754), StreamObject(objectId=13955, objectSize=45961028, streamId=191, startOffset=4367717, endOffset=4378775), StreamObject(objectId=13961, objectSize=49633652, streamId=228, startOffset=4345324, endOffset=4357266), StreamObject(objectId=13958, objectSize=47955242, streamId=221, startOffset=4276305, endOffset=4287845), StreamObject(objectId=13962, objectSize=51763218, streamId=231, startOffset=4322174, endOffset=4334636), StreamObject(objectId=13956, objectSize=52516844, streamId=206, startOffset=4373785, endOffset=4386425), StreamObject(objectId=13959, objectSize=50708485, streamId=224, startOffset=4355737, endOffset=4367946)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:24,597] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 13963 success, compacted objects: [], stream objects: [StreamObject(objectId=13964, objectSize=54187228, streamId=177, startOffset=4358208, endOffset=4371236), StreamObject(objectId=13965, objectSize=54488620, streamId=183, startOffset=4298355, endOffset=4311467), StreamObject(objectId=13970, objectSize=54715647, streamId=222, startOffset=4284765, endOffset=4297928), StreamObject(objectId=13968, objectSize=55681576, streamId=219, startOffset=4286050, endOffset=4299450), StreamObject(objectId=13972, objectSize=51519829, streamId=227, startOffset=4266784, endOffset=4279177), StreamObject(objectId=13971, objectSize=48837876, streamId=225, startOffset=4272376, endOffset=4284120), StreamObject(objectId=13967, objectSize=52944011, streamId=212, startOffset=4328703, endOffset=4341436), StreamObject(objectId=13966, objectSize=56625598, streamId=209, startOffset=4352017, endOffset=4365639), StreamObject(objectId=13973, objectSize=52751108, streamId=230, startOffset=4287714, endOffset=4300402), StreamObject(objectId=13969, objectSize=57251282, streamId=220, startOffset=4327433, endOffset=4341209)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:26,429] INFO [Controller 0] [CommitStreamObject]: stream object: 13950 commit success, compacted objects: [2588, 2625, 2656, 2669, 2704, 2739, 2774, 2787, 2822, 2857, 2908, 2925, 2960, 2995, 3008, 3043, 3078, 3113, 3126, 3162, 3197, 3221, 3256, 3279, 3314, 3349, 3373, 3408, 3443, 3478, 3501, 3525, 3560, 3595, 3630, 3665, 3700, 3724, 3747, 3782, 3806, 3841, 3876, 3911, 3946, 3981, 4005, 4028, 4063, 4098, 4122, 4157, 4192, 4227, 4257, 4274, 4309, 4344, 4379, 4414, 4438, 4473, 4508, 4531, 4555, 4590, 4625, 4660, 4695, 4730, 4742, 4777, 4812, 4847, 4871, 4906, 4941, 4976, 4999, 5023, 5058, 5093, 5128, 5163, 5187, 5222, 5245, 5280, 5304, 5339, 5376, 5411, 5446, 5470] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:29,629] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 156 to new start offset: 71778 with epoch: 2 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:33,187] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 13987 success, compacted objects: [], stream objects: [StreamObject(objectId=13988, objectSize=49127519, streamId=166, startOffset=4413621, endOffset=4425452), StreamObject(objectId=13996, objectSize=50041648, streamId=226, startOffset=4306754, endOffset=4318804), StreamObject(objectId=13993, objectSize=48870689, streamId=218, startOffset=4362536, endOffset=4374301), StreamObject(objectId=13989, objectSize=49858615, streamId=171, startOffset=4346322, endOffset=4358327), StreamObject(objectId=13990, objectSize=47112999, streamId=182, startOffset=4349311, endOffset=4360654), StreamObject(objectId=13998, objectSize=50711401, streamId=231, startOffset=4334636, endOffset=4346851), StreamObject(objectId=13997, objectSize=47496582, streamId=228, startOffset=4357266, endOffset=4368704), StreamObject(objectId=13995, objectSize=48480749, streamId=224, startOffset=4367946, endOffset=4379627), StreamObject(objectId=13991, objectSize=49751164, streamId=191, startOffset=4378775, endOffset=4390753), StreamObject(objectId=13994, objectSize=49919030, streamId=221, startOffset=4287845, endOffset=4299863), StreamObject(objectId=13992, objectSize=47441085, streamId=206, startOffset=4386425, endOffset=4397846)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:35,057] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 13974 success, compacted objects: [], stream objects: [StreamObject(objectId=13977, objectSize=43271028, streamId=162, startOffset=4834200, endOffset=4844688), StreamObject(objectId=13981, objectSize=46118269, streamId=211, startOffset=4841474, endOffset=4852657), StreamObject(objectId=13975, objectSize=50784629, streamId=158, startOffset=4823156, endOffset=4835471), StreamObject(objectId=13978, objectSize=45493464, streamId=174, startOffset=4854343, endOffset=4865377), StreamObject(objectId=13980, objectSize=50440676, streamId=195, startOffset=4830853, endOffset=4843085), StreamObject(objectId=13979, objectSize=54148724, streamId=192, startOffset=4853789, endOffset=4866925), StreamObject(objectId=13985, objectSize=48208215, streamId=232, startOffset=4828866, endOffset=4840557), StreamObject(objectId=13976, objectSize=45359910, streamId=160, startOffset=4855538, endOffset=4866538), StreamObject(objectId=13984, objectSize=50289136, streamId=229, startOffset=4787805, endOffset=4800001), StreamObject(objectId=13982, objectSize=48609502, streamId=217, startOffset=4756774, endOffset=4768562), StreamObject(objectId=13983, objectSize=46872210, streamId=223, startOffset=4829419, endOffset=4840785)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:36,706] INFO [Controller 0] [CommitStreamObject]: stream object: 13986 commit success, compacted objects: [2, 53, 70, 104, 138, 172, 206, 240, 274, 291, 325, 359, 393, 427, 461, 495, 529, 563, 597, 631, 665, 682, 716, 750, 784, 818, 835, 869, 903, 937, 971, 1005, 2870, 1057, 1094, 1122, 1158, 1193, 1228, 1263, 1298, 1333, 1368, 1385, 1419, 1453, 1487, 1521, 1555, 1590, 1624, 1658, 1692, 1726, 1743, 1777, 1811, 1845, 1879, 1913, 1947, 1981, 2003, 2032, 2066, 2100, 2134, 2168, 2202, 2236, 2270, 2304, 2338, 2372, 2389, 2423, 2457, 2491, 2525, 2542] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:38,523] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14011 success, compacted objects: [], stream objects: [StreamObject(objectId=14019, objectSize=55871635, streamId=225, startOffset=4284120, endOffset=4297587), StreamObject(objectId=14021, objectSize=53825937, streamId=230, startOffset=4300402, endOffset=4313381), StreamObject(objectId=14014, objectSize=51488470, streamId=209, startOffset=4365639, endOffset=4378049), StreamObject(objectId=14015, objectSize=51437203, streamId=212, startOffset=4341436, endOffset=4353829), StreamObject(objectId=14018, objectSize=53040473, streamId=222, startOffset=4297928, endOffset=4310715), StreamObject(objectId=14016, objectSize=56334181, streamId=219, startOffset=4299450, endOffset=4313021), StreamObject(objectId=14017, objectSize=50046914, streamId=220, startOffset=4341209, endOffset=4353265), StreamObject(objectId=14012, objectSize=56640871, streamId=177, startOffset=4371236, endOffset=4384899), StreamObject(objectId=14020, objectSize=53225699, streamId=227, startOffset=4279177, endOffset=4292008), StreamObject(objectId=14013, objectSize=56735815, streamId=183, startOffset=4311467, endOffset=4325146)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:46,373] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14023 success, compacted objects: [], stream objects: [StreamObject(objectId=14025, objectSize=46371945, streamId=171, startOffset=4358327, endOffset=4369518), StreamObject(objectId=14028, objectSize=47535256, streamId=206, startOffset=4397846, endOffset=4409314), StreamObject(objectId=14032, objectSize=51508874, streamId=226, startOffset=4318804, endOffset=4331232), StreamObject(objectId=14026, objectSize=50134346, streamId=182, startOffset=4360654, endOffset=4372746), StreamObject(objectId=14024, objectSize=46076981, streamId=166, startOffset=4425452, endOffset=4436569), StreamObject(objectId=14027, objectSize=45798058, streamId=191, startOffset=4390753, endOffset=4401795), StreamObject(objectId=14030, objectSize=46628868, streamId=221, startOffset=4299863, endOffset=4311113), StreamObject(objectId=14034, objectSize=47725246, streamId=231, startOffset=4346851, endOffset=4358363), StreamObject(objectId=14029, objectSize=51644808, streamId=218, startOffset=4374301, endOffset=4386759), StreamObject(objectId=14031, objectSize=55478479, streamId=224, startOffset=4379627, endOffset=4393017), StreamObject(objectId=14033, objectSize=49608431, streamId=228, startOffset=4368704, endOffset=4380672)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:48,599] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 13999 success, compacted objects: [], stream objects: [StreamObject(objectId=14001, objectSize=50220948, streamId=160, startOffset=4866538, endOffset=4878714), StreamObject(objectId=14000, objectSize=47096543, streamId=158, startOffset=4835471, endOffset=4846890), StreamObject(objectId=14003, objectSize=45516902, streamId=174, startOffset=4865377, endOffset=4876413), StreamObject(objectId=14004, objectSize=50448196, streamId=192, startOffset=4866925, endOffset=4879157), StreamObject(objectId=14005, objectSize=47394423, streamId=195, startOffset=4843085, endOffset=4854580), StreamObject(objectId=14002, objectSize=50806814, streamId=162, startOffset=4844688, endOffset=4857006), StreamObject(objectId=14007, objectSize=49344140, streamId=217, startOffset=4768562, endOffset=4780528), StreamObject(objectId=14008, objectSize=48638519, streamId=223, startOffset=4840785, endOffset=4852580), StreamObject(objectId=14010, objectSize=44840737, streamId=232, startOffset=4840557, endOffset=4851430), StreamObject(objectId=14006, objectSize=47884734, streamId=211, startOffset=4852657, endOffset=4864267), StreamObject(objectId=14009, objectSize=47154295, streamId=229, startOffset=4800001, endOffset=4811434)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:50,488] INFO [Controller 0] [CommitStreamObject]: stream object: 14022 commit success, compacted objects: [2589, 2626, 2657, 2670, 2705, 2740, 2775, 2788, 2823, 2858, 2909, 2926, 2961, 2996, 3009, 3044, 3079, 3114, 3127, 3163, 3198, 3222, 3257, 3280, 3315, 3350, 3374, 3409, 3444, 3479, 3502, 3526, 3561, 3596, 3631, 3666, 3701, 3725, 3748, 3783, 3807, 3842, 3877, 3912, 3947, 3982, 4006, 4029, 4064, 4099, 4123, 4158, 4193, 4228, 4258, 4275, 4310, 4345, 4380, 4415, 4439, 4474, 4509, 4532, 4556, 4591, 4626, 4661, 4696, 4731, 4743, 4778, 4813, 4848, 4872, 4907, 4942, 4977, 5000, 5024, 5059, 5094, 5129, 5164, 5188, 5223, 5246, 5281, 5305, 5340, 5377, 5412, 5447, 5472, 5497, 5529] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:28:54,148] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14048 success, compacted objects: [], stream objects: [StreamObject(objectId=14051, objectSize=55502333, streamId=209, startOffset=4378049, endOffset=4391400), StreamObject(objectId=14049, objectSize=50784126, streamId=177, startOffset=4384899, endOffset=4397115), StreamObject(objectId=14052, objectSize=54717309, streamId=212, startOffset=4353829, endOffset=4366996), StreamObject(objectId=14056, objectSize=56680688, streamId=225, startOffset=4297587, endOffset=4311227), StreamObject(objectId=14050, objectSize=54500024, streamId=183, startOffset=4325146, endOffset=4338254), StreamObject(objectId=14054, objectSize=55762734, streamId=220, startOffset=4353265, endOffset=4366679), StreamObject(objectId=14058, objectSize=52855553, streamId=230, startOffset=4313381, endOffset=4326102), StreamObject(objectId=14057, objectSize=53097215, streamId=227, startOffset=4292008, endOffset=4304787), StreamObject(objectId=14055, objectSize=50592662, streamId=222, startOffset=4310715, endOffset=4322879), StreamObject(objectId=14053, objectSize=54479439, streamId=219, startOffset=4313021, endOffset=4326132)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:00,891] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14035 success, compacted objects: [], stream objects: [StreamObject(objectId=14038, objectSize=46399481, streamId=162, startOffset=4857006, endOffset=4868269), StreamObject(objectId=14037, objectSize=47296346, streamId=160, startOffset=4878714, endOffset=4890198), StreamObject(objectId=14036, objectSize=55460508, streamId=158, startOffset=4846890, endOffset=4860350), StreamObject(objectId=14039, objectSize=49629018, streamId=174, startOffset=4876413, endOffset=4888459), StreamObject(objectId=14046, objectSize=46659099, streamId=232, startOffset=4851430, endOffset=4862757), StreamObject(objectId=14040, objectSize=48149651, streamId=192, startOffset=4879157, endOffset=4890846), StreamObject(objectId=14045, objectSize=47418806, streamId=229, startOffset=4811434, endOffset=4822946), StreamObject(objectId=14043, objectSize=48580174, streamId=217, startOffset=4780528, endOffset=4792316), StreamObject(objectId=14041, objectSize=48607814, streamId=195, startOffset=4854580, endOffset=4866380), StreamObject(objectId=14044, objectSize=47366317, streamId=223, startOffset=4852580, endOffset=4864075), StreamObject(objectId=14042, objectSize=45758036, streamId=211, startOffset=4864267, endOffset=4875375)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:01,019] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 175 to new start offset: 73675 with epoch: 2 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:01,553] INFO [Controller 0] CreateTopics result(s): CreatableTopic(name='__auto_balancer_metrics', numPartitions=-1, replicationFactor=1, assignments=[], configs=[CreateableTopicConfig(name='retention.ms', value='18000000'), CreateableTopicConfig(name='cleanup.policy', value='compact,delete')]): TOPIC_ALREADY_EXISTS (Topic '__auto_balancer_metrics' already exists.) (org.apache.kafka.controller.ReplicationControlManager)
[2023-10-13 14:29:01,983] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14059 success, compacted objects: [], stream objects: [StreamObject(objectId=14060, objectSize=46180537, streamId=166, startOffset=4436569, endOffset=4447684), StreamObject(objectId=14066, objectSize=50721176, streamId=221, startOffset=4311113, endOffset=4323325), StreamObject(objectId=14069, objectSize=47714676, streamId=228, startOffset=4380672, endOffset=4392152), StreamObject(objectId=14068, objectSize=44736142, streamId=226, startOffset=4331232, endOffset=4341998), StreamObject(objectId=14070, objectSize=49206545, streamId=231, startOffset=4358363, endOffset=4370204), StreamObject(objectId=14065, objectSize=50597835, streamId=218, startOffset=4386759, endOffset=4398932), StreamObject(objectId=14067, objectSize=48975476, streamId=224, startOffset=4393017, endOffset=4404807), StreamObject(objectId=14064, objectSize=53353636, streamId=206, startOffset=4409314, endOffset=4422154), StreamObject(objectId=14061, objectSize=46933285, streamId=171, startOffset=4369518, endOffset=4380809), StreamObject(objectId=14062, objectSize=54234893, streamId=182, startOffset=4372746, endOffset=4385795), StreamObject(objectId=14063, objectSize=46271097, streamId=191, startOffset=4401795, endOffset=4412930)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:03,585] INFO [Controller 0] [CommitStreamObject]: stream object: 14047 commit success, compacted objects: [3, 54, 71, 105, 139, 173, 207, 241, 275, 292, 326, 360, 394, 428, 462, 496, 530, 564, 598, 632, 666, 683, 717, 751, 785, 819, 836, 870, 904, 938, 972, 1006, 2871, 1058, 1095, 1123, 1159, 1194, 1229, 1264, 1299, 1334, 1369, 1386, 1420, 1454, 1488, 1522, 1556, 1591, 1625, 1659, 1693, 1727, 1744, 1778, 1812, 1846, 1880, 1914, 1948, 1982, 2006, 2033, 2067, 2101, 2135, 2169, 2203, 2237, 2271, 2305, 2339, 2373, 2390, 2424, 2458, 2492, 2526, 2543] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:11,141] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 169 to new start offset: 75028 with epoch: 2 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:11,261] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14084 success, compacted objects: [], stream objects: [StreamObject(objectId=14087, objectSize=51099398, streamId=209, startOffset=4391400, endOffset=4403698), StreamObject(objectId=14085, objectSize=54958280, streamId=177, startOffset=4397115, endOffset=4410335), StreamObject(objectId=14089, objectSize=54115408, streamId=219, startOffset=4326132, endOffset=4339148), StreamObject(objectId=14090, objectSize=51053582, streamId=220, startOffset=4366679, endOffset=4378957), StreamObject(objectId=14088, objectSize=55800305, streamId=212, startOffset=4366996, endOffset=4380417), StreamObject(objectId=14086, objectSize=54604027, streamId=183, startOffset=4338254, endOffset=4351385), StreamObject(objectId=14093, objectSize=50256231, streamId=227, startOffset=4304787, endOffset=4316874), StreamObject(objectId=14094, objectSize=53526326, streamId=230, startOffset=4326102, endOffset=4338978), StreamObject(objectId=14091, objectSize=56415896, streamId=222, startOffset=4322879, endOffset=4336443), StreamObject(objectId=14092, objectSize=57204119, streamId=225, startOffset=4311227, endOffset=4324984)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:14,125] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14071 success, compacted objects: [], stream objects: [StreamObject(objectId=14073, objectSize=45510356, streamId=160, startOffset=4890198, endOffset=4901242), StreamObject(objectId=14076, objectSize=50318781, streamId=192, startOffset=4890846, endOffset=4903053), StreamObject(objectId=14079, objectSize=46069199, streamId=217, startOffset=4792316, endOffset=4803493), StreamObject(objectId=14074, objectSize=48138745, streamId=162, startOffset=4868269, endOffset=4879948), StreamObject(objectId=14072, objectSize=54884366, streamId=158, startOffset=4860350, endOffset=4873668), StreamObject(objectId=14075, objectSize=45876107, streamId=174, startOffset=4888459, endOffset=4899588), StreamObject(objectId=14080, objectSize=51194254, streamId=223, startOffset=4864075, endOffset=4876497), StreamObject(objectId=14078, objectSize=45015312, streamId=211, startOffset=4875375, endOffset=4886299), StreamObject(objectId=14077, objectSize=49187494, streamId=195, startOffset=4866380, endOffset=4878314), StreamObject(objectId=14081, objectSize=49490607, streamId=229, startOffset=4822946, endOffset=4834957), StreamObject(objectId=14082, objectSize=45151027, streamId=232, startOffset=4862757, endOffset=4873712)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:16,809] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14107 success, compacted objects: [], stream objects: [StreamObject(objectId=14110, objectSize=48056050, streamId=182, startOffset=4385795, endOffset=4397361), StreamObject(objectId=14117, objectSize=47770080, streamId=228, startOffset=4392152, endOffset=4403652), StreamObject(objectId=14109, objectSize=47339683, streamId=171, startOffset=4380809, endOffset=4392208), StreamObject(objectId=14111, objectSize=52201073, streamId=191, startOffset=4412930, endOffset=4425495), StreamObject(objectId=14113, objectSize=52036278, streamId=218, startOffset=4398932, endOffset=4411458), StreamObject(objectId=14112, objectSize=43499851, streamId=206, startOffset=4422154, endOffset=4432629), StreamObject(objectId=14115, objectSize=49001393, streamId=224, startOffset=4404807, endOffset=4416610), StreamObject(objectId=14108, objectSize=53887446, streamId=166, startOffset=4447684, endOffset=4460664), StreamObject(objectId=14114, objectSize=49733365, streamId=221, startOffset=4323325, endOffset=4335298), StreamObject(objectId=14118, objectSize=46808099, streamId=231, startOffset=4370204, endOffset=4381467), StreamObject(objectId=14116, objectSize=48491779, streamId=226, startOffset=4341998, endOffset=4353681)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:16,821] INFO [Controller 0] [CommitStreamObject]: stream object: 14083 commit success, compacted objects: [2590, 2627, 2658, 2671, 2706, 2741, 2776, 2789, 2824, 2859, 2910, 2927, 2962, 2997, 3010, 3045, 3080, 3115, 3128, 3164, 3199, 3223, 3258, 3281, 3316, 3351, 3375, 3410, 3445, 3480, 3503, 3527, 3562, 3597, 3632, 3667, 3702, 3726, 3749, 3784, 3808, 3843, 3878, 3913, 3948, 3983, 4007, 4030, 4065, 4100, 4124, 4159, 4194, 4229, 4260, 4276, 4311, 4346, 4381, 4416, 4440, 4475, 4510, 4533, 4557, 4592, 4627, 4662, 4697, 4732, 4744, 4779, 4814, 4849, 4873, 4908, 4943, 4978, 5001, 5025, 5060, 5095, 5130, 5165, 5189, 5224, 5247, 5282, 5306, 5341, 5378, 5413, 5448, 5473, 5498, 5530, 5565] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:21,841] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 161 to new start offset: 73596 with epoch: 2 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:22,072] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 180 to new start offset: 74900 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:25,786] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14095 success, compacted objects: [], stream objects: [StreamObject(objectId=14096, objectSize=43380925, streamId=158, startOffset=4873668, endOffset=4884189), StreamObject(objectId=14098, objectSize=46847173, streamId=162, startOffset=4879948, endOffset=4891309), StreamObject(objectId=14097, objectSize=46233678, streamId=160, startOffset=4901242, endOffset=4912450), StreamObject(objectId=14099, objectSize=46958102, streamId=174, startOffset=4899588, endOffset=4910976), StreamObject(objectId=14100, objectSize=49817908, streamId=192, startOffset=4903053, endOffset=4915137), StreamObject(objectId=14102, objectSize=50816560, streamId=211, startOffset=4886299, endOffset=4898621), StreamObject(objectId=14106, objectSize=48510480, streamId=232, startOffset=4873712, endOffset=4885474), StreamObject(objectId=14103, objectSize=51202180, streamId=217, startOffset=4803493, endOffset=4815911), StreamObject(objectId=14104, objectSize=48655971, streamId=223, startOffset=4876497, endOffset=4888290), StreamObject(objectId=14101, objectSize=53604670, streamId=195, startOffset=4878314, endOffset=4891316), StreamObject(objectId=14105, objectSize=43524066, streamId=229, startOffset=4834957, endOffset=4845509)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:26,335] INFO [Controller 0] [CommitStreamObject]: stream object: 14119 commit success, compacted objects: [6, 57, 74, 108, 142, 176, 210, 244, 278, 295, 329, 363, 397, 431, 465, 499, 533, 567, 601, 635, 669, 686, 720, 754, 788, 822, 839, 873, 907, 941, 975, 1009, 2874, 1060, 1097, 1125, 1161, 1196, 1231, 1266, 1301, 1336, 1371, 1388, 1422, 1456, 1490, 1524, 1558, 1593, 1627, 1661, 1695, 1729, 1746, 1780, 1814, 1848, 1882, 1916, 1950, 1984, 2009, 2035, 2069, 2103, 2137, 2171, 2205, 2239, 2273, 2307, 2341, 2375, 2392, 2426, 2460, 2494, 2528, 2545] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:27,620] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 186 to new start offset: 74889 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:27,896] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14120 success, compacted objects: [], stream objects: [StreamObject(objectId=14122, objectSize=50551321, streamId=183, startOffset=4351385, endOffset=4363544), StreamObject(objectId=14123, objectSize=57224272, streamId=209, startOffset=4403698, endOffset=4417472), StreamObject(objectId=14124, objectSize=56461306, streamId=212, startOffset=4380417, endOffset=4394005), StreamObject(objectId=14121, objectSize=52858464, streamId=177, startOffset=4410335, endOffset=4423047), StreamObject(objectId=14127, objectSize=49524193, streamId=222, startOffset=4336443, endOffset=4348350), StreamObject(objectId=14126, objectSize=51540542, streamId=220, startOffset=4378957, endOffset=4391355), StreamObject(objectId=14129, objectSize=58536708, streamId=227, startOffset=4316874, endOffset=4330954), StreamObject(objectId=14128, objectSize=53874496, streamId=225, startOffset=4324984, endOffset=4337936), StreamObject(objectId=14130, objectSize=58838157, streamId=230, startOffset=4338978, endOffset=4353125), StreamObject(objectId=14125, objectSize=49621805, streamId=219, startOffset=4339148, endOffset=4351081)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:31,084] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 181 to new start offset: 75184 with epoch: 2 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:32,895] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14132 success, compacted objects: [], stream objects: [StreamObject(objectId=14135, objectSize=44772241, streamId=182, startOffset=4397361, endOffset=4408136), StreamObject(objectId=14134, objectSize=46770809, streamId=171, startOffset=4392208, endOffset=4403461), StreamObject(objectId=14139, objectSize=47201645, streamId=221, startOffset=4335298, endOffset=4346653), StreamObject(objectId=14133, objectSize=47897586, streamId=166, startOffset=4460664, endOffset=4472196), StreamObject(objectId=14143, objectSize=52734596, streamId=231, startOffset=4381467, endOffset=4394157), StreamObject(objectId=14138, objectSize=51119014, streamId=218, startOffset=4411458, endOffset=4423760), StreamObject(objectId=14140, objectSize=51770018, streamId=224, startOffset=4416610, endOffset=4429074), StreamObject(objectId=14137, objectSize=54382612, streamId=206, startOffset=4432629, endOffset=4445719), StreamObject(objectId=14136, objectSize=48428881, streamId=191, startOffset=4425495, endOffset=4437142), StreamObject(objectId=14141, objectSize=48557172, streamId=226, startOffset=4353681, endOffset=4365365), StreamObject(objectId=14142, objectSize=45276560, streamId=228, startOffset=4403652, endOffset=4414548)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:37,845] INFO [Controller 0] [CommitStreamObject]: stream object: 14131 commit success, compacted objects: [2592, 2629, 2660, 2673, 2708, 2743, 2778, 2791, 2826, 2861, 2912, 2929, 2964, 2999, 3012, 3047, 3082, 3117, 3130, 3165, 3200, 3224, 3259, 3282, 3317, 3352, 3376, 3411, 3446, 3481, 3504, 3528, 3563, 3598, 3633, 3668, 3703, 3727, 3750, 3785, 3809, 3844, 3879, 3914, 3949, 3984, 4008, 4031, 4066, 4101, 4125, 4160, 4195, 4230, 4263, 4277, 4312, 4347, 4382, 4417, 4441, 4476, 4511, 4534, 4558, 4593, 4628, 4663, 4698, 4733, 4745, 4780, 4815, 4850, 4874, 4909, 4944, 4979, 5002, 5026, 5061, 5096, 5131, 5166, 5190, 5225, 5248, 5283, 5307, 5342, 5379, 5414, 5449, 5476, 5500, 5531, 5566, 5601, 5625] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:38,843] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 200 to new start offset: 73039 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:44,723] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14157 success, compacted objects: [], stream objects: [StreamObject(objectId=14158, objectSize=52600384, streamId=177, startOffset=4423047, endOffset=4435703), StreamObject(objectId=14159, objectSize=56600308, streamId=183, startOffset=4363544, endOffset=4377154), StreamObject(objectId=14160, objectSize=53735085, streamId=209, startOffset=4417472, endOffset=4430397), StreamObject(objectId=14165, objectSize=51100743, streamId=225, startOffset=4337936, endOffset=4350227), StreamObject(objectId=14164, objectSize=52806760, streamId=222, startOffset=4348350, endOffset=4361050), StreamObject(objectId=14163, objectSize=52066921, streamId=220, startOffset=4391355, endOffset=4403878), StreamObject(objectId=14162, objectSize=55431357, streamId=219, startOffset=4351081, endOffset=4364414), StreamObject(objectId=14161, objectSize=52907317, streamId=212, startOffset=4394005, endOffset=4406730), StreamObject(objectId=14166, objectSize=53500629, streamId=227, startOffset=4330954, endOffset=4343819), StreamObject(objectId=14167, objectSize=58264433, streamId=230, startOffset=4353125, endOffset=4367146)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:46,113] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 163 to new start offset: 74027 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:47,117] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14168 success, compacted objects: [], stream objects: [StreamObject(objectId=14170, objectSize=54578433, streamId=171, startOffset=4403461, endOffset=4416608), StreamObject(objectId=14169, objectSize=49058297, streamId=166, startOffset=4472196, endOffset=4484007), StreamObject(objectId=14175, objectSize=50735810, streamId=221, startOffset=4346653, endOffset=4358869), StreamObject(objectId=14176, objectSize=50208573, streamId=224, startOffset=4429074, endOffset=4441161), StreamObject(objectId=14172, objectSize=45408046, streamId=191, startOffset=4437142, endOffset=4448072), StreamObject(objectId=14174, objectSize=49687773, streamId=218, startOffset=4423760, endOffset=4435727), StreamObject(objectId=14171, objectSize=49999469, streamId=182, startOffset=4408136, endOffset=4420173), StreamObject(objectId=14178, objectSize=49336557, streamId=228, startOffset=4414548, endOffset=4426419), StreamObject(objectId=14173, objectSize=42998196, streamId=206, startOffset=4445719, endOffset=4456075), StreamObject(objectId=14177, objectSize=47934877, streamId=226, startOffset=4365365, endOffset=4376910), StreamObject(objectId=14179, objectSize=48886703, streamId=231, startOffset=4394157, endOffset=4405930)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:47,304] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14144 success, compacted objects: [], stream objects: [StreamObject(objectId=14145, objectSize=47430741, streamId=158, startOffset=4884189, endOffset=4895692), StreamObject(objectId=14146, objectSize=49899193, streamId=160, startOffset=4912450, endOffset=4924553), StreamObject(objectId=14149, objectSize=43615598, streamId=192, startOffset=4915137, endOffset=4925711), StreamObject(objectId=14148, objectSize=49276236, streamId=174, startOffset=4910976, endOffset=4922928), StreamObject(objectId=14147, objectSize=48310870, streamId=162, startOffset=4891309, endOffset=4903021), StreamObject(objectId=14154, objectSize=50767583, streamId=229, startOffset=4845509, endOffset=4857822), StreamObject(objectId=14152, objectSize=44573756, streamId=217, startOffset=4815911, endOffset=4826721), StreamObject(objectId=14151, objectSize=49148635, streamId=211, startOffset=4898621, endOffset=4910538), StreamObject(objectId=14153, objectSize=49333079, streamId=223, startOffset=4888290, endOffset=4900255), StreamObject(objectId=14155, objectSize=47028640, streamId=232, startOffset=4885474, endOffset=4896882), StreamObject(objectId=14150, objectSize=50084795, streamId=195, startOffset=4891316, endOffset=4903463)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:48,337] INFO [Controller 0] [CommitStreamObject]: stream object: 14156 commit success, compacted objects: [11, 62, 79, 113, 147, 181, 215, 249, 283, 300, 334, 368, 402, 436, 470, 504, 538, 572, 606, 640, 674, 691, 725, 759, 793, 827, 844, 878, 912, 946, 980, 1014, 2875, 1063, 1100, 1128, 1164, 1199, 1234, 1269, 1304, 1339, 1374, 1391, 1425, 1459, 1493, 1527, 1561, 1596, 1630, 1664, 1698, 1732, 1749, 1783, 1817, 1851, 1885, 1919, 1953, 1987, 2015, 2038, 2072, 2106, 2140, 2174, 2208, 2242, 2276, 2310, 2344, 2378, 2395, 2429, 2463, 2497, 2531] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:51,735] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 157 to new start offset: 74027 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:57,713] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 188 to new start offset: 78205 with epoch: 6 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:29:59,616] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14193 success, compacted objects: [], stream objects: [StreamObject(objectId=14196, objectSize=55408048, streamId=209, startOffset=4430397, endOffset=4443739), StreamObject(objectId=14197, objectSize=55256322, streamId=212, startOffset=4406730, endOffset=4420042), StreamObject(objectId=14195, objectSize=50871808, streamId=183, startOffset=4377154, endOffset=4389404), StreamObject(objectId=14199, objectSize=50086458, streamId=220, startOffset=4403878, endOffset=4415938), StreamObject(objectId=14194, objectSize=56494526, streamId=177, startOffset=4435703, endOffset=4449311), StreamObject(objectId=14198, objectSize=51702049, streamId=219, startOffset=4364414, endOffset=4376857), StreamObject(objectId=14200, objectSize=57259587, streamId=222, startOffset=4361050, endOffset=4374831), StreamObject(objectId=14201, objectSize=53429000, streamId=225, startOffset=4350227, endOffset=4363089), StreamObject(objectId=14202, objectSize=52379690, streamId=227, startOffset=4343819, endOffset=4356435), StreamObject(objectId=14203, objectSize=55932048, streamId=230, startOffset=4367146, endOffset=4380620)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:00,453] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 185 to new start offset: 75664 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:01,349] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14204 success, compacted objects: [], stream objects: [StreamObject(objectId=14206, objectSize=51691494, streamId=171, startOffset=4416608, endOffset=4429064), StreamObject(objectId=14205, objectSize=46072493, streamId=166, startOffset=4484007, endOffset=4495110), StreamObject(objectId=14209, objectSize=44549287, streamId=206, startOffset=4456075, endOffset=4466798), StreamObject(objectId=14208, objectSize=51215719, streamId=191, startOffset=4448072, endOffset=4460409), StreamObject(objectId=14207, objectSize=50208201, streamId=182, startOffset=4420173, endOffset=4432276), StreamObject(objectId=14213, objectSize=46003400, streamId=226, startOffset=4376910, endOffset=4387996), StreamObject(objectId=14211, objectSize=49337284, streamId=221, startOffset=4358869, endOffset=4370759), StreamObject(objectId=14212, objectSize=50761697, streamId=224, startOffset=4441161, endOffset=4453394), StreamObject(objectId=14210, objectSize=47925478, streamId=218, startOffset=4435727, endOffset=4447275), StreamObject(objectId=14215, objectSize=48797713, streamId=231, startOffset=4405930, endOffset=4417693), StreamObject(objectId=14214, objectSize=52146896, streamId=228, startOffset=4426419, endOffset=4438979)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:01,553] INFO [Controller 0] CreateTopics result(s): CreatableTopic(name='__auto_balancer_metrics', numPartitions=-1, replicationFactor=1, assignments=[], configs=[CreateableTopicConfig(name='retention.ms', value='18000000'), CreateableTopicConfig(name='cleanup.policy', value='compact,delete')]): TOPIC_ALREADY_EXISTS (Topic '__auto_balancer_metrics' already exists.) (org.apache.kafka.controller.ReplicationControlManager)
[2023-10-13 14:30:01,684] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 198 to new start offset: 73039 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:01,698] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14180 success, compacted objects: [], stream objects: [StreamObject(objectId=14181, objectSize=54845569, streamId=158, startOffset=4895692, endOffset=4908991), StreamObject(objectId=14188, objectSize=46901508, streamId=217, startOffset=4826721, endOffset=4838091), StreamObject(objectId=14183, objectSize=49435671, streamId=162, startOffset=4903021, endOffset=4915006), StreamObject(objectId=14185, objectSize=49293282, streamId=192, startOffset=4925711, endOffset=4937665), StreamObject(objectId=14182, objectSize=49682192, streamId=160, startOffset=4924553, endOffset=4936603), StreamObject(objectId=14191, objectSize=48858152, streamId=232, startOffset=4896882, endOffset=4908730), StreamObject(objectId=14186, objectSize=43487529, streamId=195, startOffset=4903463, endOffset=4914008), StreamObject(objectId=14184, objectSize=47905134, streamId=174, startOffset=4922928, endOffset=4934544), StreamObject(objectId=14189, objectSize=47324355, streamId=223, startOffset=4900255, endOffset=4911730), StreamObject(objectId=14190, objectSize=44971251, streamId=229, startOffset=4857822, endOffset=4868727), StreamObject(objectId=14187, objectSize=46556877, streamId=211, startOffset=4910538, endOffset=4921827)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:04,160] INFO [Controller 0] [CommitStreamObject]: stream object: 14192 commit success, compacted objects: [2573, 2593, 2630, 2661, 2674, 2709, 2744, 2779, 2792, 2827, 2862, 2913, 2930, 2965, 3000, 3013, 3048, 3083, 3118, 3131, 3166, 3201, 3225, 3260, 3283, 3318, 3353, 3377, 3412, 3447, 3482, 3505, 3529, 3564, 3599, 3634, 3669, 3704, 3728, 3751, 3786, 3810, 3845, 3880, 3915, 3950, 3985, 4009, 4032, 4067, 4102, 4126, 4161, 4196, 4231, 4264, 4278, 4313, 4348, 4383, 4418, 4442, 4477, 4512, 4535, 4559, 4594, 4629, 4664, 4699, 4734, 4746, 4781, 4816, 4851, 4875, 4910, 4945, 4980, 5003, 5027, 5062, 5097, 5132, 5167, 5191, 5226, 5249, 5284, 5308, 5343, 5380, 5415, 5450, 5477, 5503, 5532, 5567, 5602, 5626, 5661, 5696] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:12,063] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14216 success, compacted objects: [], stream objects: [StreamObject(objectId=14223, objectSize=46597617, streamId=211, startOffset=4921827, endOffset=4933136), StreamObject(objectId=14222, objectSize=46337311, streamId=195, startOffset=4914008, endOffset=4925257), StreamObject(objectId=14218, objectSize=47550197, streamId=160, startOffset=4936603, endOffset=4948146), StreamObject(objectId=14224, objectSize=53067670, streamId=217, startOffset=4838091, endOffset=4850971), StreamObject(objectId=14219, objectSize=46071239, streamId=162, startOffset=4915006, endOffset=4926193), StreamObject(objectId=14221, objectSize=49757496, streamId=192, startOffset=4937665, endOffset=4949745), StreamObject(objectId=14220, objectSize=50657787, streamId=174, startOffset=4934544, endOffset=4946845), StreamObject(objectId=14227, objectSize=50632216, streamId=232, startOffset=4908730, endOffset=4921022), StreamObject(objectId=14217, objectSize=46707670, streamId=158, startOffset=4908991, endOffset=4920331), StreamObject(objectId=14225, objectSize=45944046, streamId=223, startOffset=4911730, endOffset=4922884), StreamObject(objectId=14226, objectSize=48540473, streamId=229, startOffset=4868727, endOffset=4880510)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:12,744] INFO [Controller 0] [CommitStreamObject]: stream object: 14228 commit success, compacted objects: [12, 63, 80, 114, 148, 182, 216, 250, 284, 301, 335, 369, 403, 437, 471, 505, 539, 573, 607, 641, 675, 692, 726, 760, 794, 828, 845, 879, 913, 947, 981, 1015, 2877, 1070, 1107, 1141, 1177, 1212, 1247, 1282, 1317, 1352, 1404, 1438, 1472, 1506, 1540, 1575, 1609, 1643, 1677, 1711, 1762, 1796, 1830, 1864, 1898, 1932, 1966, 2001, 2048, 2085, 2119, 2153, 2187, 2221, 2255, 2289, 2323, 2357, 2408, 2442, 2476, 2510, 2559] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:13,907] INFO [Controller 0] [CommitStreamObject]: stream object: 14229 commit success, compacted objects: [2579, 2615] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:15,120] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14230 success, compacted objects: [], stream objects: [StreamObject(objectId=14231, objectSize=45369163, streamId=166, startOffset=4495110, endOffset=4506045), StreamObject(objectId=14235, objectSize=51247588, streamId=206, startOffset=4466798, endOffset=4479140), StreamObject(objectId=14232, objectSize=51539077, streamId=171, startOffset=4429064, endOffset=4441485), StreamObject(objectId=14233, objectSize=45976823, streamId=182, startOffset=4432276, endOffset=4443343), StreamObject(objectId=14236, objectSize=44198174, streamId=218, startOffset=4447275, endOffset=4457929), StreamObject(objectId=14234, objectSize=52693238, streamId=191, startOffset=4460409, endOffset=4473105), StreamObject(objectId=14237, objectSize=49976941, streamId=221, startOffset=4370759, endOffset=4382798), StreamObject(objectId=14240, objectSize=48937122, streamId=228, startOffset=4438979, endOffset=4450763), StreamObject(objectId=14238, objectSize=51303464, streamId=224, startOffset=4453394, endOffset=4465762), StreamObject(objectId=14241, objectSize=45988671, streamId=231, startOffset=4417693, endOffset=4428772), StreamObject(objectId=14239, objectSize=51518173, streamId=226, startOffset=4387996, endOffset=4400403)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:15,812] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14243 success, compacted objects: [], stream objects: [StreamObject(objectId=14245, objectSize=54089684, streamId=183, startOffset=4389404, endOffset=4402422), StreamObject(objectId=14244, objectSize=50102342, streamId=177, startOffset=4449311, endOffset=4461365), StreamObject(objectId=14247, objectSize=51334260, streamId=212, startOffset=4420042, endOffset=4432396), StreamObject(objectId=14248, objectSize=54782801, streamId=219, startOffset=4376857, endOffset=4390040), StreamObject(objectId=14246, objectSize=57660843, streamId=209, startOffset=4443739, endOffset=4457618), StreamObject(objectId=14249, objectSize=56142961, streamId=220, startOffset=4415938, endOffset=4429437), StreamObject(objectId=14251, objectSize=54162349, streamId=225, startOffset=4363089, endOffset=4376116), StreamObject(objectId=14253, objectSize=50046535, streamId=230, startOffset=4380620, endOffset=4392667), StreamObject(objectId=14252, objectSize=53836617, streamId=227, startOffset=4356435, endOffset=4369396), StreamObject(objectId=14250, objectSize=56816900, streamId=222, startOffset=4374831, endOffset=4388489)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:28,632] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 168 to new start offset: 83092 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:28,636] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14242 success, compacted objects: [], stream objects: [StreamObject(objectId=14254, objectSize=47701767, streamId=158, startOffset=4920331, endOffset=4931906), StreamObject(objectId=14255, objectSize=47073233, streamId=160, startOffset=4948146, endOffset=4959571), StreamObject(objectId=14261, objectSize=48893125, streamId=217, startOffset=4850971, endOffset=4862834), StreamObject(objectId=14264, objectSize=49658192, streamId=232, startOffset=4921022, endOffset=4933076), StreamObject(objectId=14258, objectSize=49360250, streamId=192, startOffset=4949745, endOffset=4961725), StreamObject(objectId=14256, objectSize=48413437, streamId=162, startOffset=4926193, endOffset=4937944), StreamObject(objectId=14259, objectSize=47668645, streamId=195, startOffset=4925257, endOffset=4936824), StreamObject(objectId=14257, objectSize=47941017, streamId=174, startOffset=4946845, endOffset=4958480), StreamObject(objectId=14262, objectSize=47042273, streamId=223, startOffset=4922884, endOffset=4934301), StreamObject(objectId=14260, objectSize=49679687, streamId=211, startOffset=4933136, endOffset=4945191), StreamObject(objectId=14263, objectSize=47416799, streamId=229, startOffset=4880510, endOffset=4892017)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:28,640] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 154 to new start offset: 83092 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:28,640] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 159 to new start offset: 82908 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:28,762] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 9 to new start offset: 139437 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:30,664] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14266 success, compacted objects: [], stream objects: [StreamObject(objectId=14269, objectSize=48463854, streamId=182, startOffset=4443343, endOffset=4455005), StreamObject(objectId=14267, objectSize=49908001, streamId=166, startOffset=4506045, endOffset=4518064), StreamObject(objectId=14276, objectSize=49015586, streamId=228, startOffset=4450763, endOffset=4462563), StreamObject(objectId=14271, objectSize=52404035, streamId=206, startOffset=4479140, endOffset=4491753), StreamObject(objectId=14268, objectSize=49718384, streamId=171, startOffset=4441485, endOffset=4453447), StreamObject(objectId=14274, objectSize=46629865, streamId=224, startOffset=4465762, endOffset=4476995), StreamObject(objectId=14275, objectSize=48845808, streamId=226, startOffset=4400403, endOffset=4412161), StreamObject(objectId=14270, objectSize=47890411, streamId=191, startOffset=4473105, endOffset=4484638), StreamObject(objectId=14273, objectSize=49599656, streamId=221, startOffset=4382798, endOffset=4394742), StreamObject(objectId=14272, objectSize=47740278, streamId=218, startOffset=4457929, endOffset=4469417), StreamObject(objectId=14277, objectSize=48642659, streamId=231, startOffset=4428772, endOffset=4440485)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:30,963] INFO [Controller 0] [CommitStreamObject]: stream object: 14265 commit success, compacted objects: [2662, 2675, 2710, 2745, 2780, 2793, 2828, 2863, 2914, 2931, 2966, 3001, 3014, 3049, 3084, 3119, 3132, 3167, 3202, 3226, 3261, 3284, 3319, 3354, 3378, 3413, 3448, 3483, 3506, 3530, 3565, 3600, 3635, 3670, 3705, 3729, 3752, 3787, 3811, 3846, 3881, 3916, 3951, 3986, 4010, 4033, 4068, 4103, 4127, 4162, 4197, 4232, 4266, 4279, 4314, 4349, 4384, 4419, 4444, 4478, 4513, 4536, 4560, 4595, 4630, 4665, 4700, 4735, 4747, 4782, 4817, 4852, 4876, 4911, 4946, 4981, 5004, 5028, 5063, 5098, 5133, 5168, 5192, 5227, 5250, 5285, 5309, 5344, 5381, 5416, 5451, 5479, 5504, 5533, 5568, 5603, 5627, 5662, 5697, 5732, 5755] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:33,313] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14290 success, compacted objects: [], stream objects: [StreamObject(objectId=14291, objectSize=51060757, streamId=177, startOffset=4461365, endOffset=4473642), StreamObject(objectId=14292, objectSize=53539329, streamId=183, startOffset=4402422, endOffset=4415297), StreamObject(objectId=14297, objectSize=50271491, streamId=220, startOffset=4429437, endOffset=4441526), StreamObject(objectId=14300, objectSize=55662770, streamId=227, startOffset=4369396, endOffset=4382778), StreamObject(objectId=14294, objectSize=61180009, streamId=212, startOffset=4432396, endOffset=4447103), StreamObject(objectId=14296, objectSize=57390566, streamId=219, startOffset=4390040, endOffset=4403838), StreamObject(objectId=14298, objectSize=49415551, streamId=222, startOffset=4388489, endOffset=4400368), StreamObject(objectId=14301, objectSize=55491114, streamId=230, startOffset=4392667, endOffset=4406013), StreamObject(objectId=14293, objectSize=52852919, streamId=209, startOffset=4457618, endOffset=4470333), StreamObject(objectId=14299, objectSize=52218585, streamId=225, startOffset=4376116, endOffset=4388671)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:34,893] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 164 to new start offset: 76750 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:39,233] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 199 to new start offset: 73644 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:42,987] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 187 to new start offset: 87153 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:42,989] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14278 success, compacted objects: [], stream objects: [StreamObject(objectId=14281, objectSize=46216980, streamId=162, startOffset=4937944, endOffset=4949160), StreamObject(objectId=14279, objectSize=46265236, streamId=158, startOffset=4931906, endOffset=4943130), StreamObject(objectId=14283, objectSize=47868850, streamId=192, startOffset=4961725, endOffset=4973341), StreamObject(objectId=14280, objectSize=52112395, streamId=160, startOffset=4959571, endOffset=4972216), StreamObject(objectId=14286, objectSize=41972786, streamId=217, startOffset=4862834, endOffset=4873016), StreamObject(objectId=14282, objectSize=50010531, streamId=174, startOffset=4958480, endOffset=4970615), StreamObject(objectId=14285, objectSize=50904929, streamId=211, startOffset=4945191, endOffset=4957540), StreamObject(objectId=14288, objectSize=50899697, streamId=229, startOffset=4892017, endOffset=4904368), StreamObject(objectId=14289, objectSize=47646051, streamId=232, startOffset=4933076, endOffset=4944635), StreamObject(objectId=14287, objectSize=44033339, streamId=223, startOffset=4934301, endOffset=4944984), StreamObject(objectId=14284, objectSize=53059740, streamId=195, startOffset=4936824, endOffset=4949696)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:43,055] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 172 to new start offset: 90769 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:43,056] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 155 to new start offset: 83527 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:43,724] INFO [Controller 0] [CommitStreamObject]: stream object: 14295 commit success, compacted objects: [15, 66, 83, 117, 151, 185, 219, 253, 287, 304, 338, 372, 406, 440, 474, 508, 542, 576, 610, 644, 678, 695, 729, 763, 797, 831, 848, 882, 916, 950, 984, 1018, 2905, 1071, 1108, 1142, 1178, 1213, 1248, 1283, 1318, 1353, 1405, 1439, 1473, 1507, 1541, 1576, 1610, 1644, 1678, 1712, 1763, 1797, 1831, 1865, 1899, 1933, 1967, 2004, 2050, 2086, 2120, 2154, 2188, 2222, 2256, 2290, 2324, 2358, 2409, 2443, 2477, 2511, 2560] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:45,860] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14303 success, compacted objects: [], stream objects: [StreamObject(objectId=14304, objectSize=51210206, streamId=166, startOffset=4518064, endOffset=4530402), StreamObject(objectId=14305, objectSize=51024193, streamId=171, startOffset=4453447, endOffset=4465730), StreamObject(objectId=14307, objectSize=48441983, streamId=191, startOffset=4484638, endOffset=4496299), StreamObject(objectId=14321, objectSize=48119162, streamId=231, startOffset=4440485, endOffset=4452071), StreamObject(objectId=14306, objectSize=46707169, streamId=182, startOffset=4455005, endOffset=4466252), StreamObject(objectId=14308, objectSize=50602726, streamId=206, startOffset=4491753, endOffset=4503935), StreamObject(objectId=14313, objectSize=44718407, streamId=221, startOffset=4394742, endOffset=4405507), StreamObject(objectId=14314, objectSize=49853479, streamId=224, startOffset=4476995, endOffset=4489008), StreamObject(objectId=14318, objectSize=52721216, streamId=228, startOffset=4462563, endOffset=4475251), StreamObject(objectId=14317, objectSize=47289106, streamId=226, startOffset=4412161, endOffset=4423545), StreamObject(objectId=14311, objectSize=48147521, streamId=218, startOffset=4469417, endOffset=4481010)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:46,902] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 167 to new start offset: 73039 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:46,930] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 202 to new start offset: 84697 with epoch: 2 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:46,960] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 250 to new start offset: 816 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:46,973] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 249 to new start offset: 143138129 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:47,098] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14302 success, compacted objects: [], stream objects: [StreamObject(objectId=14310, objectSize=47625720, streamId=160, startOffset=4972216, endOffset=4983788), StreamObject(objectId=14309, objectSize=48891188, streamId=158, startOffset=4943130, endOffset=4955010), StreamObject(objectId=14323, objectSize=50689095, streamId=223, startOffset=4944984, endOffset=4957303), StreamObject(objectId=14312, objectSize=52606787, streamId=162, startOffset=4949160, endOffset=4961943), StreamObject(objectId=14324, objectSize=47671941, streamId=229, startOffset=4904368, endOffset=4915953), StreamObject(objectId=14315, objectSize=43408726, streamId=174, startOffset=4970615, endOffset=4981161), StreamObject(objectId=14316, objectSize=50871720, streamId=192, startOffset=4973341, endOffset=4985703), StreamObject(objectId=14322, objectSize=52362555, streamId=217, startOffset=4873016, endOffset=4885739), StreamObject(objectId=14320, objectSize=44034724, streamId=211, startOffset=4957540, endOffset=4968242), StreamObject(objectId=14325, objectSize=46146918, streamId=232, startOffset=4944635, endOffset=4955849), StreamObject(objectId=14319, objectSize=49854954, streamId=195, startOffset=4949696, endOffset=4961810)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:47,168] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 201 to new start offset: 84697 with epoch: 2 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:47,751] INFO [Controller 0] [CommitStreamObject]: stream object: 14326 commit success, compacted objects: [2580, 2616] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:49,794] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 193 to new start offset: 74494 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:50,235] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14327 success, compacted objects: [], stream objects: [StreamObject(objectId=14328, objectSize=54673499, streamId=177, startOffset=4473642, endOffset=4486791), StreamObject(objectId=14329, objectSize=50978408, streamId=183, startOffset=4415297, endOffset=4427559), StreamObject(objectId=14330, objectSize=55903212, streamId=209, startOffset=4470333, endOffset=4483779), StreamObject(objectId=14332, objectSize=52827411, streamId=219, startOffset=4403838, endOffset=4416545), StreamObject(objectId=14331, objectSize=52202606, streamId=212, startOffset=4447103, endOffset=4459661), StreamObject(objectId=14333, objectSize=53992099, streamId=220, startOffset=4441526, endOffset=4454505), StreamObject(objectId=14335, objectSize=58080522, streamId=225, startOffset=4388671, endOffset=4402645), StreamObject(objectId=14336, objectSize=51452115, streamId=227, startOffset=4382778, endOffset=4395163), StreamObject(objectId=14334, objectSize=53256274, streamId=222, startOffset=4400368, endOffset=4413180), StreamObject(objectId=14337, objectSize=55620094, streamId=230, startOffset=4406013, endOffset=4419395)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:30:53,148] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 184 to new start offset: 74120 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:00,986] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14339 success, compacted objects: [], stream objects: [StreamObject(objectId=14340, objectSize=50684012, streamId=166, startOffset=4530402, endOffset=4542606), StreamObject(objectId=14341, objectSize=52588101, streamId=171, startOffset=4465730, endOffset=4478385), StreamObject(objectId=14343, objectSize=47064081, streamId=191, startOffset=4496299, endOffset=4507622), StreamObject(objectId=14345, objectSize=51543395, streamId=218, startOffset=4481010, endOffset=4493413), StreamObject(objectId=14342, objectSize=49147852, streamId=182, startOffset=4466252, endOffset=4478076), StreamObject(objectId=14346, objectSize=45074790, streamId=221, startOffset=4405507, endOffset=4416359), StreamObject(objectId=14348, objectSize=45159176, streamId=226, startOffset=4423545, endOffset=4434413), StreamObject(objectId=14347, objectSize=48448626, streamId=224, startOffset=4489008, endOffset=4500670), StreamObject(objectId=14344, objectSize=46241985, streamId=206, startOffset=4503935, endOffset=4515060), StreamObject(objectId=14349, objectSize=51163701, streamId=228, startOffset=4475251, endOffset=4487570), StreamObject(objectId=14350, objectSize=51789226, streamId=231, startOffset=4452071, endOffset=4464537)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:01,553] INFO [Controller 0] CreateTopics result(s): CreatableTopic(name='__auto_balancer_metrics', numPartitions=-1, replicationFactor=1, assignments=[], configs=[CreateableTopicConfig(name='retention.ms', value='18000000'), CreateableTopicConfig(name='cleanup.policy', value='compact,delete')]): TOPIC_ALREADY_EXISTS (Topic '__auto_balancer_metrics' already exists.) (org.apache.kafka.controller.ReplicationControlManager)
[2023-10-13 14:31:03,001] INFO [Controller 0] [TrimStream]: node: 4 trim stream: 196 to new start offset: 73037 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:06,676] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14351 success, compacted objects: [], stream objects: [StreamObject(objectId=14353, objectSize=55478797, streamId=183, startOffset=4427559, endOffset=4440896), StreamObject(objectId=14354, objectSize=57300980, streamId=209, startOffset=4483779, endOffset=4497565), StreamObject(objectId=14356, objectSize=53796378, streamId=219, startOffset=4416545, endOffset=4429485), StreamObject(objectId=14352, objectSize=55686120, streamId=177, startOffset=4486791, endOffset=4500193), StreamObject(objectId=14355, objectSize=51681836, streamId=212, startOffset=4459661, endOffset=4472095), StreamObject(objectId=14360, objectSize=48519694, streamId=227, startOffset=4395163, endOffset=4406837), StreamObject(objectId=14359, objectSize=52914714, streamId=225, startOffset=4402645, endOffset=4415377), StreamObject(objectId=14358, objectSize=53239758, streamId=222, startOffset=4413180, endOffset=4425982), StreamObject(objectId=14361, objectSize=56344204, streamId=230, startOffset=4419395, endOffset=4432955), StreamObject(objectId=14357, objectSize=54025381, streamId=220, startOffset=4454505, endOffset=4467502)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:07,122] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 178 to new start offset: 73644 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:09,624] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 153 to new start offset: 82908 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:09,659] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 9 to new start offset: 140847 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:11,512] INFO [Controller 0] [CommitStreamObject]: stream object: 14338 commit success, compacted objects: [2663, 2676, 2711, 2746, 2781, 2794, 2829, 2864, 2915, 2932, 2967, 3002, 3015, 3050, 3085, 3120, 3133, 3168, 3203, 3227, 3262, 3285, 3320, 3355, 3379, 3414, 3449, 3484, 3507, 3531, 3566, 3601, 3636, 3671, 3706, 3730, 3753, 3788, 3812, 3847, 3882, 3917, 3952, 3987, 4011, 4034, 4069, 4104, 4128, 4163, 4198, 4233, 4268, 4280, 4315, 4350, 4385, 4420, 4447, 4479, 4514, 4537, 4561, 4596, 4631, 4666, 4701, 4736, 4748, 4783, 4818, 4853, 4877, 4912, 4947, 4982, 5005, 5029, 5064, 5099, 5134, 5169, 5193, 5228, 5251, 5286, 5310, 5345, 5382, 5417, 5452, 5481, 5511, 5534, 5569, 5604, 5628, 5663, 5698, 5733, 5756, 5782, 5817] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:14,914] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14393 success, compacted objects: [], stream objects: [StreamObject(objectId=14396, objectSize=47148535, streamId=182, startOffset=4478076, endOffset=4489431), StreamObject(objectId=14394, objectSize=50974876, streamId=166, startOffset=4542606, endOffset=4554894), StreamObject(objectId=14398, objectSize=49696173, streamId=206, startOffset=4515060, endOffset=4527035), StreamObject(objectId=14397, objectSize=52383764, streamId=191, startOffset=4507622, endOffset=4520240), StreamObject(objectId=14401, objectSize=45953481, streamId=224, startOffset=4500670, endOffset=4511741), StreamObject(objectId=14395, objectSize=44356323, streamId=171, startOffset=4478385, endOffset=4489068), StreamObject(objectId=14403, objectSize=50361587, streamId=228, startOffset=4487570, endOffset=4499697), StreamObject(objectId=14402, objectSize=48961598, streamId=226, startOffset=4434413, endOffset=4446211), StreamObject(objectId=14400, objectSize=49735874, streamId=221, startOffset=4416359, endOffset=4428339), StreamObject(objectId=14404, objectSize=47727121, streamId=231, startOffset=4464537, endOffset=4476034), StreamObject(objectId=14399, objectSize=51443563, streamId=218, startOffset=4493413, endOffset=4505804)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:15,861] INFO [Controller 0] [CommitWALObject]: node: 0 commit wal object: 14362 success, compacted objects: [], stream objects: [StreamObject(objectId=14372, objectSize=48490496, streamId=162, startOffset=4961943, endOffset=4973725), StreamObject(objectId=14377, objectSize=44321572, streamId=217, startOffset=4885739, endOffset=4896509), StreamObject(objectId=14376, objectSize=50109056, streamId=211, startOffset=4968242, endOffset=4980416), StreamObject(objectId=14371, objectSize=51828874, streamId=160, startOffset=4983788, endOffset=4996382), StreamObject(objectId=14378, objectSize=47185733, streamId=223, startOffset=4957303, endOffset=4968768), StreamObject(objectId=14374, objectSize=48149249, streamId=192, startOffset=4985703, endOffset=4997404), StreamObject(objectId=14375, objectSize=50981490, streamId=195, startOffset=4961810, endOffset=4974200), StreamObject(objectId=14370, objectSize=47238534, streamId=158, startOffset=4955010, endOffset=4966488), StreamObject(objectId=14379, objectSize=51167029, streamId=229, startOffset=4915953, endOffset=4928386), StreamObject(objectId=14380, objectSize=44380327, streamId=232, startOffset=4955849, endOffset=4966634), StreamObject(objectId=14373, objectSize=50079883, streamId=174, startOffset=4981161, endOffset=4993330)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:16,470] INFO [Controller 0] [CommitStreamObject]: stream object: 14392 commit success, compacted objects: [18, 35, 86, 120, 154, 188, 222, 256, 307, 341, 375, 409, 443, 477, 511, 545, 579, 613, 647, 698, 732, 766, 800, 851, 885, 919, 953, 987, 1022] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:19,933] INFO [Controller 0] [TrimStream]: node: 3 trim stream: 179 to new start offset: 83297 with epoch: 4 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:20,272] INFO [Controller 0] [CommitWALObject]: node: 3 commit wal object: 14406 success, compacted objects: [], stream objects: [StreamObject(objectId=14407, objectSize=54181106, streamId=177, startOffset=4500193, endOffset=4513279), StreamObject(objectId=14409, objectSize=55252573, streamId=209, startOffset=4497565, endOffset=4510910), StreamObject(objectId=14414, objectSize=51625353, streamId=225, startOffset=4415377, endOffset=4427844), StreamObject(objectId=14415, objectSize=56222980, streamId=227, startOffset=4406837, endOffset=4420411), StreamObject(objectId=14408, objectSize=51412896, streamId=183, startOffset=4440896, endOffset=4453312), StreamObject(objectId=14410, objectSize=52586237, streamId=212, startOffset=4472095, endOffset=4484798), StreamObject(objectId=14416, objectSize=53957804, streamId=230, startOffset=4432955, endOffset=4445979), StreamObject(objectId=14411, objectSize=52029147, streamId=219, startOffset=4429485, endOffset=4442050), StreamObject(objectId=14412, objectSize=54202101, streamId=220, startOffset=4467502, endOffset=4480593), StreamObject(objectId=14413, objectSize=56861511, streamId=222, startOffset=4425982, endOffset=4439711)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:22,430] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 89 to new start offset: 30021597 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:22,432] INFO [Controller 0] [TrimStream]: node: 0 trim stream: 143 to new start offset: 10572672 with epoch: 0 success (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:23,342] INFO [Controller 0] [CommitStreamObject]: stream object: 14405 commit success, compacted objects: [1041, 1077, 1110, 1144, 1180, 1215, 1250, 1285, 1320, 1355, 1407, 1441, 1475, 1509, 1543, 1578, 1612, 1646, 1680, 1714, 1765, 1799, 1833, 1867, 1901, 1935, 1969, 2007, 2054, 2088, 2122, 2156, 2190, 2224, 2258, 2292, 2326, 2360, 2411, 2445, 2479, 2513, 2562, 2575, 2581, 2617] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:29,506] INFO [Controller 0] [CommitWALObject]: node: 4 commit wal object: 14418 success, compacted objects: [], stream objects: [StreamObject(objectId=14420, objectSize=50512324, streamId=171, startOffset=4489068, endOffset=4501266), StreamObject(objectId=14419, objectSize=44741511, streamId=166, startOffset=4554894, endOffset=4565693), StreamObject(objectId=14422, objectSize=49465136, streamId=191, startOffset=4520240, endOffset=4532166), StreamObject(objectId=14421, objectSize=48670713, streamId=182, startOffset=4489431, endOffset=4501176), StreamObject(objectId=14427, objectSize=49348977, streamId=226, startOffset=4446211, endOffset=4458118), StreamObject(objectId=14424, objectSize=53790939, streamId=218, startOffset=4505804, endOffset=4518783), StreamObject(objectId=14428, objectSize=47297359, streamId=228, startOffset=4499697, endOffset=4511118), StreamObject(objectId=14425, objectSize=51333112, streamId=221, startOffset=4428339, endOffset=4440727), StreamObject(objectId=14423, objectSize=46738826, streamId=206, startOffset=4527035, endOffset=4538313), StreamObject(objectId=14426, objectSize=47771531, streamId=224, startOffset=4511741, endOffset=4523270), StreamObject(objectId=14429, objectSize=48803166, streamId=231, startOffset=4476034, endOffset=4487800)] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:36,838] INFO [Controller 0] [CommitStreamObject]: stream object: 14417 commit success, compacted objects: [2664, 2677, 2712, 2747, 2782, 2795, 2830, 2865, 2916, 2933, 2968, 3003, 3016, 3051, 3086, 3121, 3134, 3169, 3204, 3228, 3263, 3286, 3321, 3356, 3380, 3415, 3450, 3485, 3508, 3532, 3567, 3602, 3637, 3672, 3707, 3731, 3754, 3789, 3813, 3848, 3883, 3918, 3953, 3988, 4012, 4035, 4070, 4105, 4129, 4164, 4199, 4234, 4269, 4281, 4316, 4351, 4386, 4421, 4449, 4480, 4515, 4538, 4562, 4597, 4632, 4667, 4702, 4737, 4749, 4784, 4819, 4854, 4878, 4913, 4948, 4983, 5006, 5030, 5065, 5100, 5135, 5170, 5194, 5229, 5252, 5287, 5311, 5346, 5383, 5418, 5453, 5483, 5512, 5535, 5570, 5605, 5629, 5664, 5699, 5734, 5757, 5783, 5818, 5853, 5888, 5923] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:40,621] INFO [Controller 0] [CommitStreamObject]: stream object: 14430 commit success, compacted objects: [24, 41, 92, 126, 160, 194, 228, 262, 313, 347, 381, 415, 449, 483, 517, 551, 585, 619, 653, 704, 738, 772, 806, 857, 891, 925, 959, 993, 1028] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:46,561] INFO [Controller 0] [CommitStreamObject]: stream object: 14431 commit success, compacted objects: [1045, 1081, 2879, 1133, 1169, 1204, 1239, 1274, 1309, 1344, 1379, 1396, 1430, 1464, 1498, 1532, 1566, 1601, 1635, 1669, 1703, 1737, 1754, 1788, 1822, 1856, 1890, 1924, 1958, 1992, 2026, 2047, 2077, 2111, 2145, 2179, 2213, 2247, 2281, 2315, 2349, 2383, 2400, 2434, 2468, 2502, 2536, 2551] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:31:59,689] INFO [Controller 0] [CommitStreamObject]: stream object: 14432 commit success, compacted objects: [2594, 2631, 2665, 2678, 2713, 2748, 2783, 2796, 2831, 2866, 2917, 2934, 2969, 3004, 3017, 3052, 3087, 3122, 3135, 3170, 3205, 3229, 3264, 3287, 3322, 3357, 3381, 3416, 3451, 3486, 3509, 3533, 3568, 3603, 3638, 3673, 3708, 3732, 3755, 3790, 3814, 3849, 3884, 3919, 3954, 3989, 4013, 4036, 4071, 4106, 4130, 4165, 4200, 4235, 4270, 4282, 4317, 4352, 4387, 4422, 4450, 4481, 4516, 4539, 4563, 4598, 4633, 4668, 4703, 4738, 4750, 4785, 4820, 4855, 4879, 4914, 4949, 4984, 5007, 5031, 5066, 5101, 5136, 5171, 5195, 5230, 5253, 5288, 5312, 5347, 5384, 5419, 5454, 5485, 5513, 5536, 5571, 5606, 5630, 5665, 5700, 5735, 5758, 5784, 5819, 5854, 5889, 5924, 5948, 5983] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:01,553] INFO [Controller 0] CreateTopics result(s): CreatableTopic(name='__auto_balancer_metrics', numPartitions=-1, replicationFactor=1, assignments=[], configs=[CreateableTopicConfig(name='retention.ms', value='18000000'), CreateableTopicConfig(name='cleanup.policy', value='compact,delete')]): TOPIC_ALREADY_EXISTS (Topic '__auto_balancer_metrics' already exists.) (org.apache.kafka.controller.ReplicationControlManager)
[2023-10-13 14:32:03,264] INFO [Controller 0] [CommitStreamObject]: stream object: 14433 commit success, compacted objects: [30, 47, 98, 132, 166, 200, 234, 268, 319, 353, 387, 421, 455, 489, 523, 557, 591, 625, 659, 710, 744, 778, 812, 863, 897, 931, 965, 999, 1034] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:07,730] INFO [Controller 0] [CommitStreamObject]: stream object: 14434 commit success, compacted objects: [1075, 1117, 1151, 1187, 1222, 1257, 1292, 1327, 1362, 1414, 1448, 1482, 1516, 1550, 1585, 1619, 1653, 1687, 1721, 1772, 1806, 1840, 1874, 1908, 1942, 1976, 2021, 2061, 2095, 2129, 2163, 2197, 2231, 2265, 2299, 2333, 2367, 2418, 2452, 2486, 2520, 2569] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:08,807] INFO [Controller 0] [CommitStreamObject]: stream object: 14435 commit success, compacted objects: [2587, 2622] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:21,702] INFO [Controller 0] [CommitStreamObject]: stream object: 14436 commit success, compacted objects: [2666, 2679, 2714, 2749, 2784, 2797, 2832, 2867, 2918, 2935, 2970, 3005, 3018, 3053, 3088, 3123, 3136, 3171, 3206, 3230, 3265, 3288, 3323, 3358, 3382, 3417, 3452, 3487, 3510, 3534, 3569, 3604, 3639, 3674, 3709, 3733, 3757, 3791, 3815, 3850, 3885, 3920, 3955, 3990, 4014, 4037, 4072, 4107, 4131, 4166, 4201, 4236, 4271, 4283, 4318, 4353, 4388, 4423, 4452, 4482, 4517, 4540, 4564, 4599, 4634, 4669, 4704, 4739, 4751, 4786, 4821, 4856, 4880, 4915, 4950, 4985, 5008, 5032, 5067, 5102, 5137, 5172, 5196, 5231, 5254, 5289, 5313, 5348, 5385, 5420, 5455, 5487, 5514, 5537, 5572, 5607, 5631, 5666, 5701, 5736, 5759, 5785, 5820, 5855, 5890, 5925, 5949, 5984, 6019] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:25,378] INFO [Controller 0] [CommitStreamObject]: stream object: 14437 commit success, compacted objects: [33, 50, 101, 135, 169, 203, 237, 271, 322, 356, 390, 424, 458, 492, 526, 560, 594, 628, 663, 713, 747, 781, 815, 866, 900, 934, 968, 1002, 1037] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:26,352] INFO [Controller 0] [CommitStreamObject]: stream object: 14438 commit success, compacted objects: [1053, 1089] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:32,058] INFO [Controller 0] [CommitStreamObject]: stream object: 14439 commit success, compacted objects: [2923, 1136, 1172, 1207, 1242, 1277, 1312, 1347, 1382, 1399, 1433, 1467, 1501, 1535, 1569, 1604, 1638, 1672, 1706, 1740, 1757, 1791, 1825, 1859, 1893, 1927, 1961, 1995, 2029, 2053, 2080, 2114, 2148, 2182, 2216, 2250, 2284, 2318, 2352, 2386, 2403, 2437, 2471, 2505, 2539, 2554] (org.apache.kafka.controller.stream.StreamControlManager)
[2023-10-13 14:32:44,354] INFO [Controller 0] [CommitStreamObject]: stream object: 14440 commit success, compacted objects: [2595, 2632, 2667, 2680, 2715, 2750, 2785, 2798, 2833, 2868, 2919, 2936, 2971, 3006, 3019, 3054, 3089, 3124, 3137, 3172, 3207, 3231, 3266, 3289, 3324, 3359, 3383, 3418, 3453, 3488, 3511, 3535, 3570, 3605, 3640, 3675, 3710, 3734, 3759, 3792, 3816, 3851, 3886, 3921, 3956, 3991, 4015, 4038, 4073, 4108, 4132, 4167, 4202, 4237, 4272, 4284, 4319, 4354, 4389, 4424, 4454, 4483, 4518, 4541, 4565, 4600, 4635, 4670, 4705, 4740, 4752, 4787, 4822, 4857, 4881, 4916, 4951, 4986, 5009, 5033, 5068, 5103, 5138, 5173, 5197, 5232, 5255, 5290, 5314, 5349, 5386, 5421, 5456, 5489, 5515, 5538, 5573, 5608, 5632, 5667, 5702, 5737, 5760, 5786, 5821, 5856, 5891, 5926, 5950, 5985, 6020, 6043, 6067] (org.apache.kafka.controller.stream.StreamControlManager)
Chillax-0v0 commented 1 year ago

producer:

14:28:09.939 [main] INFO WorkloadGenerator - Pub rate 25568.664 msg/s / 99.878 MB/s | Pub err     0.000 err/s | Cons rate 25566.087 msg/s / 99.867 MB/s | Backlog:  0.017 K | Pub Latency (ms) avg:  7.803 - 50%:  3.104 - 99%: 84.532 - 99.9%: 145.617 - Max
: 199.994 | Pub Delay Latency (us) avg: 57.897 - 50%: 56.000 - 99%: 66.000 - 99.9%: 2000.000 - Max: 14110.000 | E2E Latency (ms) avg:  9.864 - 50%:  4.643 - 99%: 90.984 - 99.9%: 148.046 - Max: 208.156
14:28:20.020 [main] INFO WorkloadGenerator - Pub rate 25633.241 msg/s / 100.130 MB/s | Pub err     0.000 err/s | Cons rate 25636.514 msg/s / 100.143 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  5.713 - 50%:  2.916 - 99%: 39.927 - 99.9%: 136.688 - M
ax: 149.287 | Pub Delay Latency (us) avg: 58.888 - 50%: 56.000 - 99%: 66.000 - 99.9%: 2527.000 - Max: 7411.000 | E2E Latency (ms) avg:  7.459 - 50%:  4.360 - 99%: 44.136 - 99.9%: 138.970 - Max: 152.308
14:28:30.104 [main] INFO WorkloadGenerator - Pub rate 23853.437 msg/s / 93.177 MB/s | Pub err     0.000 err/s | Cons rate 23837.866 msg/s / 93.117 MB/s | Backlog:  0.141 K | Pub Latency (ms) avg:  7.665 - 50%:  2.712 - 99%: 70.714 - 99.9%: 130.451 - Max
: 151.101 | Pub Delay Latency (us) avg: 57.744 - 50%: 56.000 - 99%: 66.000 - 99.9%: 1867.000 - Max: 9928.000 | E2E Latency (ms) avg: 10.229 - 50%:  4.152 - 99%: 93.932 - 99.9%: 151.384 - Max: 176.236
14:28:40.225 [main] INFO WorkloadGenerator - Pub rate 27297.402 msg/s / 106.630 MB/s | Pub err     0.000 err/s | Cons rate 27314.927 msg/s / 106.699 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg: 681.998 - 50%:  3.577 - 99%: 7804.991 - 99.9%: 7978.527
 - Max: 8150.975 | Pub Delay Latency (us) avg: 18448.609 - 50%: 56.000 - 99%: 476839.000 - 99.9%: 551115.000 - Max: 553963.000 | E2E Latency (ms) avg: 687.476 - 50%:  5.212 - 99%: 7821.407 - 99.9%: 8028.991 - Max: 8147.743
14:28:50.343 [main] INFO WorkloadGenerator - Pub rate 25629.105 msg/s / 100.114 MB/s | Pub err     0.000 err/s | Cons rate 25626.538 msg/s / 100.104 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg: 99.776 - 50%:  2.798 - 99%: 2183.375 - 99.9%: 2417.151
- Max: 2447.711 | Pub Delay Latency (us) avg: 60.322 - 50%: 56.000 - 99%: 66.000 - 99.9%: 3165.000 - Max: 9856.000 | E2E Latency (ms) avg: 101.210 - 50%:  4.280 - 99%: 2184.159 - 99.9%: 2422.607 - Max: 2452.127
14:29:00.441 [main] INFO WorkloadGenerator - Pub rate 25595.224 msg/s / 99.981 MB/s | Pub err     0.000 err/s | Cons rate 25596.609 msg/s / 99.987 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  6.471 - 50%:  2.930 - 99%: 58.120 - 99.9%: 129.004 - Max
: 149.444 | Pub Delay Latency (us) avg: 60.974 - 50%: 56.000 - 99%: 67.000 - 99.9%: 3718.000 - Max: 10164.000 | E2E Latency (ms) avg:  8.459 - 50%:  4.332 - 99%: 67.732 - 99.9%: 131.547 - Max: 152.241
14:29:10.537 [main] INFO WorkloadGenerator - Pub rate 25587.254 msg/s / 99.950 MB/s | Pub err     0.000 err/s | Cons rate 25585.371 msg/s / 99.943 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg: 16.598 - 50%:  3.130 - 99%: 447.423 - 99.9%: 614.775 - Ma
x: 641.811 | Pub Delay Latency (us) avg: 57.397 - 50%: 56.000 - 99%: 66.000 - 99.9%: 1842.000 - Max: 7644.000 | E2E Latency (ms) avg: 18.831 - 50%:  4.604 - 99%: 455.243 - 99.9%: 617.599 - Max: 642.299
14:29:20.638 [main] INFO WorkloadGenerator - Pub rate 25610.293 msg/s / 100.040 MB/s | Pub err     0.000 err/s | Cons rate 25612.867 msg/s / 100.050 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg: 19.999 - 50%:  2.874 - 99%: 578.411 - 99.9%: 746.251 -
Max: 845.219 | Pub Delay Latency (us) avg: 61.216 - 50%: 56.000 - 99%: 66.000 - 99.9%: 3840.000 - Max: 8767.000 | E2E Latency (ms) avg: 21.934 - 50%:  4.226 - 99%: 580.683 - 99.9%: 748.867 - Max: 837.295
14:29:30.735 [main] INFO WorkloadGenerator - Pub rate 25580.236 msg/s / 99.923 MB/s | Pub err     0.000 err/s | Cons rate 25551.440 msg/s / 99.810 MB/s | Backlog:  0.260 K | Pub Latency (ms) avg:  4.853 - 50%:  2.948 - 99%: 30.226 - 99.9%: 88.495 - Max:
 120.711 | Pub Delay Latency (us) avg: 57.437 - 50%: 56.000 - 99%: 65.000 - 99.9%: 1962.000 - Max: 8472.000 | E2E Latency (ms) avg:  6.601 - 50%:  4.378 - 99%: 36.371 - 99.9%: 95.555 - Max: 120.310
14:29:40.815 [main] INFO WorkloadGenerator - Pub rate 25623.391 msg/s / 100.091 MB/s | Pub err     0.000 err/s | Cons rate 25650.677 msg/s / 100.198 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  6.992 - 50%:  2.897 - 99%: 64.881 - 99.9%: 97.504 - Ma
x: 123.633 | Pub Delay Latency (us) avg: 60.428 - 50%: 56.000 - 99%: 66.000 - 99.9%: 3475.000 - Max: 9239.000 | E2E Latency (ms) avg:  9.279 - 50%:  4.273 - 99%: 75.723 - 99.9%: 109.236 - Max: 140.213
14:29:50.927 [main] INFO WorkloadGenerator - Pub rate 25557.082 msg/s / 99.832 MB/s | Pub err     0.000 err/s | Cons rate 25530.139 msg/s / 99.727 MB/s | Backlog:  0.257 K | Pub Latency (ms) avg: 478.618 - 50%:  3.796 - 99%: 5848.735 - 99.9%: 6024.991 -
 Max: 6055.103 | Pub Delay Latency (us) avg: 314.180 - 50%: 55.000 - 99%: 1159.000 - 99.9%: 56379.000 - Max: 64944.000 | E2E Latency (ms) avg: 608.699 - 50%:  7.845 - 99%: 6687.327 - 99.9%: 7132.639 - Max: 7248.511
14:30:01.021 [main] INFO WorkloadGenerator - Pub rate 23978.680 msg/s / 93.667 MB/s | Pub err     0.000 err/s | Cons rate 23993.515 msg/s / 93.725 MB/s | Backlog:  0.107 K | Pub Latency (ms) avg:  6.329 - 50%:  2.842 - 99%: 65.589 - 99.9%: 129.509 - Max
: 135.468 | Pub Delay Latency (us) avg: 63.887 - 50%: 56.000 - 99%: 66.000 - 99.9%: 4817.000 - Max: 12980.000 | E2E Latency (ms) avg:  8.022 - 50%:  4.217 - 99%: 67.543 - 99.9%: 130.976 - Max: 137.562
14:30:11.124 [main] INFO WorkloadGenerator - Pub rate 27249.978 msg/s / 106.445 MB/s | Pub err     0.000 err/s | Cons rate 27262.368 msg/s / 106.494 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg: 201.831 - 50%:  2.783 - 99%: 3571.903 - 99.9%: 3749.487
 - Max: 3827.487 | Pub Delay Latency (us) avg: 58.236 - 50%: 56.000 - 99%: 67.000 - 99.9%: 2150.000 - Max: 10422.000 | E2E Latency (ms) avg: 203.415 - 50%:  4.159 - 99%: 3578.367 - 99.9%: 3752.303 - Max: 3815.487
14:30:21.223 [main] INFO WorkloadGenerator - Pub rate 25621.566 msg/s / 100.084 MB/s | Pub err     0.000 err/s | Cons rate 25622.653 msg/s / 100.088 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  7.397 - 50%:  3.154 - 99%: 68.322 - 99.9%: 126.676 - M
ax: 142.149 | Pub Delay Latency (us) avg: 59.597 - 50%: 56.000 - 99%: 66.000 - 99.9%: 3321.000 - Max: 11917.000 | E2E Latency (ms) avg:  9.192 - 50%:  4.579 - 99%: 72.372 - 99.9%: 132.119 - Max: 145.039
14:30:31.320 [main] INFO WorkloadGenerator - Pub rate 25565.190 msg/s / 99.864 MB/s | Pub err     0.000 err/s | Cons rate 25556.466 msg/s / 99.830 MB/s | Backlog:  0.059 K | Pub Latency (ms) avg: 95.496 - 50%:  3.544 - 99%: 1809.383 - 99.9%: 1958.423 -
Max: 1982.607 | Pub Delay Latency (us) avg: 57.299 - 50%: 55.000 - 99%: 66.000 - 99.9%: 1369.000 - Max: 8271.000 | E2E Latency (ms) avg: 99.078 - 50%:  5.138 - 99%: 1814.215 - 99.9%: 1966.295 - Max: 1987.135
14:30:41.420 [main] INFO WorkloadGenerator - Pub rate 24620.714 msg/s / 96.175 MB/s | Pub err     0.000 err/s | Cons rate 24619.131 msg/s / 96.168 MB/s | Backlog:  0.075 K | Pub Latency (ms) avg:  9.499 - 50%:  2.886 - 99%: 84.526 - 99.9%: 141.202 - Max
: 179.200 | Pub Delay Latency (us) avg: 59.588 - 50%: 56.000 - 99%: 66.000 - 99.9%: 2292.000 - Max: 9551.000 | E2E Latency (ms) avg: 12.375 - 50%:  4.340 - 99%: 96.060 - 99.9%: 160.883 - Max: 206.474
14:30:51.526 [main] INFO WorkloadGenerator - Pub rate 26594.986 msg/s / 103.887 MB/s | Pub err     0.000 err/s | Cons rate 26601.128 msg/s / 103.911 MB/s | Backlog:  0.013 K | Pub Latency (ms) avg: 149.534 - 50%:  4.398 - 99%: 2752.639 - 99.9%: 2922.719
 - Max: 2952.143 | Pub Delay Latency (us) avg: 60.453 - 50%: 55.000 - 99%: 67.000 - 99.9%: 3622.000 - Max: 12037.000 | E2E Latency (ms) avg: 152.949 - 50%:  6.227 - 99%: 2762.479 - 99.9%: 2929.647 - Max: 2963.471
14:31:01.624 [main] INFO WorkloadGenerator - Pub rate 25603.379 msg/s / 100.013 MB/s | Pub err     0.000 err/s | Cons rate 25590.619 msg/s / 99.963 MB/s | Backlog:  0.142 K | Pub Latency (ms) avg: 10.866 - 50%:  3.321 - 99%: 86.568 - 99.9%: 176.944 - Ma
x: 189.714 | Pub Delay Latency (us) avg: 58.685 - 50%: 56.000 - 99%: 66.000 - 99.9%: 2360.000 - Max: 8150.000 | E2E Latency (ms) avg: 13.751 - 50%:  4.856 - 99%: 97.249 - 99.9%: 180.276 - Max: 195.517
14:31:11.711 [main] INFO WorkloadGenerator - Pub rate 23214.193 msg/s / 90.680 MB/s | Pub err     0.000 err/s | Cons rate 23217.366 msg/s / 90.693 MB/s | Backlog:  0.110 K | Pub Latency (ms) avg:  7.394 - 50%:  2.610 - 99%: 69.964 - 99.9%: 110.129 - Max
: 127.501 | Pub Delay Latency (us) avg: 63.500 - 50%: 56.000 - 99%: 67.000 - 99.9%: 4391.000 - Max: 14223.000 | E2E Latency (ms) avg:  9.495 - 50%:  3.935 - 99%: 76.153 - 99.9%: 115.700 - Max: 551.367
14:31:21.837 [main] INFO WorkloadGenerator - Pub rate 27957.072 msg/s / 109.207 MB/s | Pub err     0.000 err/s | Cons rate 27969.740 msg/s / 109.257 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg: 763.405 - 50%:  5.825 - 99%: 8376.351 - 99.9%: 8586.687
 - Max: 8609.599 | Pub Delay Latency (us) avg: 204868.401 - 50%: 56.000 - 99%: 1894375.000 - 99.9%: 1975375.000 - Max: 1982567.000 | E2E Latency (ms) avg: 826.384 - 50%:  9.442 - 99%: 8503.359 - 99.9%: 9136.255 - Max: 9371.327
14:31:31.943 [main] INFO WorkloadGenerator - Pub rate 22888.586 msg/s / 89.409 MB/s | Pub err     0.000 err/s | Cons rate 22888.586 msg/s / 89.409 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  5.479 - 50%:  2.462 - 99%: 51.947 - 99.9%: 110.475 - Max
: 127.708 | Pub Delay Latency (us) avg: 60.437 - 50%: 56.000 - 99%: 65.000 - 99.9%: 2981.000 - Max: 11852.000 | E2E Latency (ms) avg:  7.153 - 50%:  3.766 - 99%: 57.114 - 99.9%: 116.511 - Max: 130.533
14:31:42.007 [main] INFO WorkloadGenerator - Pub rate  2144.363 msg/s /  8.376 MB/s | Pub err     0.000 err/s | Cons rate  2145.257 msg/s /  8.380 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  2.324 - 50%:  1.697 - 99%: 14.084 - 99.9%: 21.512 - Max:
 24.656 | Pub Delay Latency (us) avg: 86.227 - 50%: 55.000 - 99%: 1846.000 - 99.9%: 3443.000 - Max: 10302.000 | E2E Latency (ms) avg:  3.941 - 50%:  2.991 - 99%: 26.955 - 99.9%: 34.251 - Max: 35.249
14:31:52.117 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:
  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:32:02.236 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:
  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:32:02.417 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Disconnecting from node 0 due to request timeout.
14:32:02.418 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Cancelled in-flight PRODUCE request with correlation id 14281617 due to node 0 being disconnected (elapsed time since creation: 36055ms, elapse
d time since send: 36055ms, request timeout: 30000ms)
14:32:02.418 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Cancelled in-flight PRODUCE request with correlation id 14281618 due to node 0 being disconnected (elapsed time since creation: 36055ms, elapse
d time since send: 36055ms, request timeout: 30000ms)
14:32:02.418 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Cancelled in-flight PRODUCE request with correlation id 14281619 due to node 0 being disconnected (elapsed time since creation: 36055ms, elapse
d time since send: 36055ms, request timeout: 30000ms)
14:32:02.418 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Cancelled in-flight PRODUCE request with correlation id 14281620 due to node 0 being disconnected (elapsed time since creation: 36055ms, elapse
d time since send: 36055ms, request timeout: 30000ms)
14:32:02.418 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Cancelled in-flight PRODUCE request with correlation id 14281621 due to node 0 being disconnected (elapsed time since creation: 36054ms, elapsed time since send: 36054ms, request timeout: 30000ms)
14:32:02.418 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Got error produce response with correlation id 14281617 on topic-partition test-topic-0000000-1, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Received invalid metadata error in produce request on partition test-topic-0000000-1 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Got error produce response with correlation id 14281618 on topic-partition test-topic-0000000-1, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Received invalid metadata error in produce request on partition test-topic-0000000-1 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Got error produce response with correlation id 14281619 on topic-partition test-topic-0000000-1, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Received invalid metadata error in produce request on partition test-topic-0000000-1 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Got error produce response with correlation id 14281620 on topic-partition test-topic-0000000-1, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Received invalid metadata error in produce request on partition test-topic-0000000-1 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Got error produce response with correlation id 14281621 on topic-partition test-topic-0000000-1, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.419 [kafka-producer-network-thread | producer-7] WARN Sender - [Producer clientId=producer-7] Received invalid metadata error in produce request on partition test-topic-0000000-1 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.665 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Disconnecting from node 0 due to request timeout.
14:32:02.666 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Cancelled in-flight PRODUCE request with correlation id 13655804 due to node 0 being disconnected (elapsed time since creation: 36304ms, elapsed time since send: 36304ms, request timeout: 30000ms)
14:32:02.666 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Cancelled in-flight PRODUCE request with correlation id 13655805 due to node 0 being disconnected (elapsed time since creation: 36303ms, elapsed time since send: 36303ms, request timeout: 30000ms)
14:32:02.666 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Cancelled in-flight PRODUCE request with correlation id 13655806 due to node 0 being disconnected (elapsed time since creation: 36303ms, elapsed time since send: 36303ms, request timeout: 30000ms)
14:32:02.666 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Cancelled in-flight PRODUCE request with correlation id 13655807 due to node 0 being disconnected (elapsed time since creation: 36303ms, elapsed time since send: 36303ms, request timeout: 30000ms)
14:32:02.666 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Cancelled in-flight PRODUCE request with correlation id 13655808 due to node 0 being disconnected (elapsed time since creation: 36302ms, elapsed time since send: 36302ms, request timeout: 30000ms)
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Got error produce response with correlation id 13655804 on topic-partition test-topic-0000000-27, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Received invalid metadata error in produce request on partition test-topic-0000000-27 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Got error produce response with correlation id 13655805 on topic-partition test-topic-0000000-27, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Received invalid metadata error in produce request on partition test-topic-0000000-27 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Got error produce response with correlation id 13655806 on topic-partition test-topic-0000000-27, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Received invalid metadata error in produce request on partition test-topic-0000000-27 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Got error produce response with correlation id 13655807 on topic-partition test-topic-0000000-27, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Received invalid metadata error in produce request on partition test-topic-0000000-27 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.666 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Got error produce response with correlation id 13655808 on topic-partition test-topic-0000000-27, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.667 [kafka-producer-network-thread | producer-5] WARN Sender - [Producer clientId=producer-5] Received invalid metadata error in produce request on partition test-topic-0000000-27 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.793 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Disconnecting from node 0 due to request timeout.
14:32:02.794 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Cancelled in-flight PRODUCE request with correlation id 13915510 due to node 0 being disconnected (elapsed time since creation: 36434ms, elapsed time since send: 36434ms, request timeout: 30000ms)
14:32:02.794 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Cancelled in-flight PRODUCE request with correlation id 13915511 due to node 0 being disconnected (elapsed time since creation: 36433ms, elapsed time since send: 36433ms, request timeout: 30000ms)
14:32:02.794 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Cancelled in-flight PRODUCE request with correlation id 13915512 due to node 0 being disconnected (elapsed time since creation: 36432ms, elapsed time since send: 36432ms, request timeout: 30000ms)
14:32:02.794 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Cancelled in-flight PRODUCE request with correlation id 13915513 due to node 0 being disconnected (elapsed time since creation: 36431ms, elapsed time since send: 36431ms, request timeout: 30000ms)
14:32:02.794 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Cancelled in-flight PRODUCE request with correlation id 13915514 due to node 0 being disconnected (elapsed time since creation: 36430ms, elapsed time since send: 36430ms, request timeout: 30000ms)
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Got error produce response with correlation id 13915510 on topic-partition test-topic-0000000-24, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Received invalid metadata error in produce request on partition test-topic-0000000-24 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Got error produce response with correlation id 13915511 on topic-partition test-topic-0000000-24, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Received invalid metadata error in produce request on partition test-topic-0000000-24 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Got error produce response with correlation id 13915512 on topic-partition test-topic-0000000-24, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Received invalid metadata error in produce request on partition test-topic-0000000-24 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Got error produce response with correlation id 13915513 on topic-partition test-topic-0000000-24, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Received invalid metadata error in produce request on partition test-topic-0000000-24 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Got error produce response with correlation id 13915514 on topic-partition test-topic-0000000-24, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.794 [kafka-producer-network-thread | producer-8] WARN Sender - [Producer clientId=producer-8] Received invalid metadata error in produce request on partition test-topic-0000000-24 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.873 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Disconnecting from node 0 due to request timeout.
14:32:02.873 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Cancelled in-flight PRODUCE request with correlation id 13983706 due to node 0 being disconnected (elapsed time since creation: 36501ms, elapsed time since send: 36501ms, request timeout: 30000ms)
14:32:02.873 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Cancelled in-flight PRODUCE request with correlation id 13983707 due to node 0 being disconnected (elapsed time since creation: 36500ms, elapsed time since send: 36500ms, request timeout: 30000ms)
14:32:02.873 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Cancelled in-flight PRODUCE request with correlation id 13983708 due to node 0 being disconnected (elapsed time since creation: 36500ms, elapsed time since send: 36500ms, request timeout: 30000ms)
14:32:02.873 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Cancelled in-flight PRODUCE request with correlation id 13983709 due to node 0 being disconnected (elapsed time since creation: 36500ms, elapsed time since send: 36500ms, request timeout: 30000ms)
14:32:02.873 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Cancelled in-flight PRODUCE request with correlation id 13983710 due to node 0 being disconnected (elapsed time since creation: 36500ms, elapsed time since send: 36500ms, request timeout: 30000ms)
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Got error produce response with correlation id 13983706 on topic-partition test-topic-0000000-12, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Received invalid metadata error in produce request on partition test-topic-0000000-12 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Got error produce response with correlation id 13983707 on topic-partition test-topic-0000000-12, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Received invalid metadata error in produce request on partition test-topic-0000000-12 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Got error produce response with correlation id 13983708 on topic-partition test-topic-0000000-12, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Received invalid metadata error in produce request on partition test-topic-0000000-12 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Got error produce response with correlation id 13983709 on topic-partition test-topic-0000000-12, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Received invalid metadata error in produce request on partition test-topic-0000000-12 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Got error produce response with correlation id 13983710 on topic-partition test-topic-0000000-12, retrying (2147483646 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 0
14:32:02.874 [kafka-producer-network-thread | producer-6] WARN Sender - [Producer clientId=producer-6] Received invalid metadata error in produce request on partition test-topic-0000000-12 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. Going to request metadata update now
14:32:12.358 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:32:22.477 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:32:32.390 [local-worker-2-2] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:813) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803) ~[?:?]
        at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195) ~[?:?]
        at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:47) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$6(LocalWorker.java:206) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$7(LocalWorker.java:204) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.65.Final.jar:4.1.65.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
14:32:32.545 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Disconnecting from node 0 due to request timeout.
14:32:32.546 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Cancelled in-flight API_VERSIONS request with correlation id 14294769 due to node 0 being disconnected (elapsed time since creation: 30085ms, elapsed time since send: 30085ms, request timeout: 30000ms)
14:32:32.561 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.099 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:32:32.638 [local-worker-2-4] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:813) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803) ~[?:?]
        at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195) ~[?:?]
        at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:47) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$6(LocalWorker.java:206) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$7(LocalWorker.java:204) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.65.Final.jar:4.1.65.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
14:32:32.749 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Disconnecting from node 0 due to request timeout.
14:32:32.749 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Cancelled in-flight API_VERSIONS request with correlation id 13670055 due to node 0 being disconnected (elapsed time since creation: 30027ms, elapsed time since send: 30027ms, request timeout: 30000ms)
14:32:32.765 [local-worker-2-3] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:813) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803) ~[?:?]
        at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195) ~[?:?]
        at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:47) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$6(LocalWorker.java:206) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$7(LocalWorker.java:204) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.65.Final.jar:4.1.65.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
14:32:32.844 [local-worker-2-1] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:813) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803) ~[?:?]
        at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195) ~[?:?]
        at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:47) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$6(LocalWorker.java:206) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$7(LocalWorker.java:204) ~[io.openmessaging.benchmark-benchmark-framework-0.0.1-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.65.Final.jar:4.1.65.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate 131072 bytes within the configured max blocking time 60000 ms. Total memory: 33554432 bytes. Available memory: 0 bytes. Poolable size: 131072 bytes
14:32:32.873 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Disconnecting from node 0 due to request timeout.
14:32:32.873 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Cancelled in-flight API_VERSIONS request with correlation id 13929852 due to node 0 being disconnected (elapsed time since creation: 30029ms, elapsed time since send: 30029ms, request timeout: 30000ms)
14:32:32.946 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Disconnecting from node 0 due to request timeout.
14:32:32.947 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Cancelled in-flight API_VERSIONS request with correlation id 14000115 due to node 0 being disconnected (elapsed time since creation: 30028ms, elapsed time since send: 30028ms, request timeout: 30000ms)
14:32:39.617 [kafka-admin-client-thread | adminclient-2] INFO NetworkClient - [AdminClient clientId=adminclient-2] Node 4 disconnected.
14:32:42.656 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.297 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:32:52.778 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:33:02.693 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Disconnecting from node 0 due to request timeout.
14:33:02.693 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Cancelled in-flight API_VERSIONS request with correlation id 14294771 due to node 0 being disconnected (elapsed time since creation: 30029ms, elapsed time since send: 30029ms, request timeout: 30000ms)
14:33:02.862 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:33:02.897 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Disconnecting from node 0 due to request timeout.
14:33:02.897 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Cancelled in-flight API_VERSIONS request with correlation id 13670057 due to node 0 being disconnected (elapsed time since creation: 30028ms, elapsed time since send: 30028ms, request timeout: 30000ms)
14:33:03.001 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Disconnecting from node 0 due to request timeout.
14:33:03.001 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Cancelled in-flight API_VERSIONS request with correlation id 13929854 due to node 0 being disconnected (elapsed time since creation: 30027ms, elapsed time since send: 30027ms, request timeout: 30000ms)
14:33:03.069 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Disconnecting from node 0 due to request timeout.
14:33:03.069 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Cancelled in-flight API_VERSIONS request with correlation id 14000117 due to node 0 being disconnected (elapsed time since creation: 30028ms, elapsed time since send: 30028ms, request timeout: 30000ms)
14:33:09.645 [kafka-admin-client-thread | adminclient-2] INFO AdminMetadataManager - [AdminClient clientId=adminclient-2] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
14:33:12.944 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:33:23.062 [main] INFO WorkloadGenerator - Pub rate     0.000 msg/s /  0.000 MB/s | Pub err     0.000 err/s | Cons rate     0.000 msg/s /  0.000 MB/s | Backlog:  0.000 K | Pub Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | Pub Delay Latency (us) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000 | E2E Latency (ms) avg:  0.000 - 50%:  0.000 - 99%:  0.000 - 99.9%:  0.000 - Max:  0.000
14:33:25.055 [kafka-producer-network-thread | producer-7] INFO NetworkClient - [Producer clientId=producer-7] Node 4 disconnected.
14:33:25.056 [kafka-producer-network-thread | producer-8] INFO NetworkClient - [Producer clientId=producer-8] Node 4 disconnected.
14:33:25.056 [kafka-producer-network-thread | producer-5] INFO NetworkClient - [Producer clientId=producer-5] Node 4 disconnected.
14:33:25.057 [kafka-producer-network-thread | producer-6] INFO NetworkClient - [Producer clientId=producer-6] Node 4 disconnected.
14:33:26.362 [kafka-producer-network-thread | producer-8] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for test-topic-0000000-24:120002 ms has passed since batch creation
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkProducer.lambda$sendAsync$0(KafkaBenchmarkProducer.java:56) ~[io.openmessaging.benchmark-driver-kafka-0.0.1-SNAPSHOT.jar:?]
        at org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:392) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for test-topic-0000000-24:120002 ms has passed since batch creation
14:33:26.362 [kafka-producer-network-thread | producer-8] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for test-topic-0000000-24:120002 ms has passed since batch creation
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkProducer.lambda$sendAsync$0(KafkaBenchmarkProducer.java:56) ~[io.openmessaging.benchmark-driver-kafka-0.0.1-SNAPSHOT.jar:?]
        at org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:392) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for test-topic-0000000-24:120002 ms has passed since batch creation
14:33:26.363 [kafka-producer-network-thread | producer-5] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkProducer.lambda$sendAsync$0(KafkaBenchmarkProducer.java:56) ~[io.openmessaging.benchmark-driver-kafka-0.0.1-SNAPSHOT.jar:?]
        at org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:392) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
14:33:26.363 [kafka-producer-network-thread | producer-5] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkProducer.lambda$sendAsync$0(KafkaBenchmarkProducer.java:56) ~[io.openmessaging.benchmark-driver-kafka-0.0.1-SNAPSHOT.jar:?]
        at org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:392) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
14:33:26.363 [kafka-producer-network-thread | producer-5] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkProducer.lambda$sendAsync$0(KafkaBenchmarkProducer.java:56) ~[io.openmessaging.benchmark-driver-kafka-0.0.1-SNAPSHOT.jar:?]
        at org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:392) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
14:33:26.364 [kafka-producer-network-thread | producer-5] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
        at io.openmessaging.benchmark.driver.kafka.KafkaBenchmarkProducer.lambda$sendAsync$0(KafkaBenchmarkProducer.java:56) ~[io.openmessaging.benchmark-driver-kafka-0.0.1-SNAPSHOT.jar:?]
        at org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:766) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:751) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:392) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [org.apache.kafka-kafka-clients-3.4.0.jar:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 4 record(s) for test-topic-0000000-27:120002 ms has passed since batch creation
superhx commented 1 year ago
superhx commented 1 year ago

If components exceed 16, CompositeByteBuffer will invoke #consolidateIfNeeded allocating memory to merge exceed components. image

SCNieh commented 1 year ago

resolved in https://github.com/AutoMQ/rocketmq-on-s3/pull/361/