Open mlowicki opened 6 months ago
It happens because if _GNU_SOURCE
is defined, it's using strndup
here
https://github.com/confluentinc/librdkafka/blob/2587cac70f83fced42c51f921bed325a434f5bc7/src/rdkafka_request.c#L1236
that stops at first NULL byte differently from the alternative implementation librdkafka provides. Needs to be fixed
Description
I was reproducing with code in Rust using
rust-rdkafka
but that wrapper doesn't do much while reading committed offsets besides calling librdkafka itself. Because inrust-rdkafka
it checks that metadata is a valid UTF-8 string it panics with errors like:if it starts to return "random" data.
Verified also by implementing
OffsetFetch
andOffsetCommit
in Rust that it's not an issue on the Kafka side - with pure Rust impl I couldn't reproduce issue with reading invalid data.How to reproduce
Use byte array
[10, 20, 0, 30, 40]
as the commit metadata and commit for any partition. Then read committed offsets viard_kafka_committed
and in some cases metadata after \0 is just different than what was written.Examples from other tests I've conducted where for the same metadata written we get random responses:
rust-rdkafka
usedlibrdkafka
2.3.0 - https://github.com/fede1024/rust-rdkafka/commit/87105bcb44e37bb35bf3eabdf92ef73b9a3d2c18.Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
<REPLACE with e.g., v0.10.5 or a git sha. NOT "latest" or "current">
3.7.0
<REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
this is all I set:
32-Ubuntu SMP Mon Jan 9 12:28:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
debug=..
as necessary) from librdkafkaNothing is logged and everything seems to be working just fine.
Can't do it but no errors / warning on the broker side. Also as said above I've configured it isn't the issue purely on the Kafka side.