search

IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.44k stars 1.75k forks source link

fix: allow skipping per message size check #2892

Closed ae-govau closed 3 weeks ago

ae-govau commented 4 months ago

This is useful when producer compression is enabled, as it allows for much larger messages to be sent to the broker (providing the entire batch they are in stays small enough).

The previous workaround suggested in #2142 causes other ill-effects related to message batching (such as in #2797).

Fixes #2142, probably also #2797 and may help mitigate not having merged #2851.

It's not clear how easy it is to write a test for this but happy to try if we're happy enough with this approach.

ae-govau commented 4 months ago

This still doesn't solve everything - https://github.com/IBM/sarama/blob/0ab2bb77aeca321f41a0953a8c6f52472607a59e/produce_set.go#L253-L255 still is checking a before-compression amount for calculating the batch, but I think the above is probably better than erroneously returning errors for small messages when MaxMessageBytes is set too high.

puellanivis commented 4 months ago

The phrasing of “skip” here is potentially confusing, especially when batch processing. Not sure there’s a better phrasing possible though.

PS: Maybe Disable?

github-actions[bot] commented 1 month ago

Thank you for your contribution! However, this pull request has not had any activity in the past 90 days and will be closed in 30 days if no updates occur. If you believe the changes are still valid then please verify your branch has no conflicts with main and rebase if needed. If you are awaiting a (re-)review then please let us know.