search

wbarnha / kafka-python-ng

Fork for Python client for Apache Kafka
https://wbarnha.github.io/kafka-python-ng/
Apache License 2.0
77 stars 10 forks source link

zstd Compression Does not work with the Consumer #78

Open wbarnha opened 8 months ago

wbarnha commented 8 months ago

We recently switched over to using zstd and have some monitoring, and other tools that use this library. After updating to 2.0.2 we where disappointed to find out that it doesn't work with our data.

The problem seems to be that no matter what version you configure in the consumer it only sends FetchRequest_V4 as per this logic and FetchRequest_v10 is required for zstd and your own note

Sending FetchRequest_V4 against a topic that has zstd data returns this error:

DEBUG:kafka.protocol.parser:Processing response FetchResponse_v4
DEBUG:kafka.conn:<BrokerConnection node_id=0 host=localhost:9092 <connected> [IPv4 ('127.0.0.1', 9092)]> Response 4302 (1.6283988952636719 ms): FetchResponse_v4(throttle_time_ms=0, topics=[(topics='foobar', partitions=[(partition=0, error_code=76, highwater_offset=-1, last_stable_offset=-1, aborted_transactions=NULL, message_set=b'')])])

Namely the error_code=76 which is UNSUPPORTED_COMPRESSION_TYPE

trim21 commented 1 month ago

do you accept a PR?

wbarnha commented 1 month ago

I do! I've been busy these days so I'm not actively writing or reviewing code. Once things calm down, I'll get back on the horse. 😀 Please feel free to open a PR though, if it's small, I'll merge it!

trim21 commented 1 month ago

do you mind I skip request version 4-9 and implement 10 directly on supported kafka version？

wbarnha commented 1 month ago

Go for it!

trim21 commented 1 month ago

sorry, I just find that confluent-kafka-python support this, I'll just use confluent-kafka-python....

trim21 commented 1 month ago

looks like to support kafka v10， we'll need to support session first…

trim21 commented 1 month ago

OK, good new is, session is optional feature and can be omitted.

trim21 commented 1 month ago

https://github.com/wbarnha/kafka-python-ng/pull/200

I'm not sure it's a big or small PR, but it would works fine to send basic v10 request.

New feature can be omitted by sending zero value.