maxBatchSize preallocated memory may be thousands of times larger than actual message length

[keyboardbobo]

Is your enhancement request related to a problem? Please describe. Guys, during the stress test, we found that the client's memory is very large, and even Full gc appeared. After analyzing the dump file, we found that the memory space occupied is much larger than the actual message size, and the serialization of 1KB of messages takes up 1MB.

batchedMessageMetadataAndPayload = PulsarByteBufAllocator.DEFAULT
                        .buffer(Math.min(maxBatchSize, ClientCnx.getMaxMessageSize()));

maxBatchSize = Math.max(maxBatchSize, uncompressedSize);

Debugging found that the maxBatchSize that controls the pre-allocated ByteBufPair.b2 memory size is stateful. As the size of the largest batch or the largest single message grows, this may cause the pre-allocated ByteBufPair.b2 memory to grow larger and larger , which maybe thousands of times larger than the payload of MessageImpl.

Lowering the value of batchingMaxMessagesmay reduce the risk of problems, but a single message that may be too large can also cause problems

Describe the solution you'd like It is best to loop through the messages to be packed to accurately calculate the memory size to be allocated

Describe alternatives you've considered Let the user choose whether to precisely allocate or pre-allocate

Additional context

maxPendingMessages=2000
maxPendingMessagesAcrossPartitions=40000
blockIfQueueFull=false
sendTimeoutMs=5000
batchingMaxPublishDelayMicros=50
batchingMaxMessages=2000
batchingMaxBytes=5242880
batchingEnabled=true

[gaozhangmin]

I have a doubt, How to achieve precisely allocate? @keyboardbobo

[keyboardbobo]

It is best to loop through the messages to be packed to accurately calculate the memory size to be allocated

Whether it is possible to loop through the messages to be packaged and add their sizes to accurately calculate the size of the memory to be allocated？

[tjiuming]

How about use CompositeByteBuf, allocate small memory and let it grow. @codelipenghui Could you please assign it to me?

[codelipenghui]

@tjiuming Yes, I think CompositeByteBuf will work which provided a Bytebuf that can be dynamically expanded

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.