search

tulios / kafkajs

A modern Apache Kafka client for node.js
https://kafka.js.org
MIT License
3.75k stars 527 forks source link

Spliting `sendBatch` produce calls to `message.max.bytes` per partition #1227

Open MartinBurian opened 3 years ago

MartinBurian commented 3 years ago

Is your feature request related to a problem? Please describe. When KafkaJS sends a batch of messages to Kafka, the broker accepts at most max.message.bytesB of messages per partition (defaults to broker's message.max.bytes, 1MB). The current sendBatch API does not take this into account, and will send all the messages it received and possibly fail. When it fails, the error says "The request included a message larger than the max message size the server will accept", which is not true. Each message is smaller than the limit, only all the messages combined by partition are over the limit.

Describe the solution you'd like In my opinion, KafkaJS should take the limit into account, and split the partition record batch if it exceeds the limit. The limit should be provided as a configuration parameter of the producer, and default to 1MB.

The app using KafkaJS cannot work around this at 100% efficiency, because the limit is enforced at partition level, and partitons are assigned inside KafkaJS. The app can split the message batch it sends via sendBatch into 1MB chunks, but then it won't fill the produce call as much as possible, introducing potentially unnecessary calls.

I'd be happy to contribute the solution, if it is considered helpful. Although it may take me some time...

Additional context The issue has been discussed in https://github.com/tulios/kafkajs/issues/262. Other client libraries don't usually hit this limit, because they buffer the messages produced to partition, and send them in batches of at least 1 message and at most batch.sizeB. This config happens to default to 1MB, which is exactly the default message.max.bytes (thus triggering this error if and only if there is a single message larger than the max.message.bytes limit).

HippolyteQuief commented 2 years ago

👍

gabegorelick commented 1 year ago

This has also been discussed in https://github.com/tulios/kafkajs/issues/949.

leppaott commented 11 months ago

Would be nice as can't be avoided except by increasing the server values..