BewareMyPower
commented
2 years ago What's your broker's config? I tried to reproduce this issue with a KoP standalone but it works well.
Here are my configs in conf/standalone.conf:
messagingProtocols=kafka
kafkaListeners=PLAINTEXT://127.0.0.1:9092
brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendBrokerTimestampMetadataInterceptor,org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
brokerDeleteInactiveTopicsEnabled=false
# It should be noted that the latest Kafka clients requires transaction enabled
brokerDeduplicationEnabled=true
kafkaTransactionCoordinatorEnabled=trueThen, since rdkafka disables topic auto creation by default, you must create the topic in advance:
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-001 --partitions 1 --replication-factor 1Then I ran the following Python script:
from confluent_kafka import Consumer
c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'my-kafka-group', 'auto.offset.reset': 'earliest'})
c.subscribe(['test-001'])
while True:
msg = c.poll(1.0)
if msg is None:
continue
if msg.error():
print("Consumer error: {}".format(msg.error()))
continue
print('Received message: {}'.format(msg.value().decode('utf-8')))
c.close()It could receive the messages produced by the Kafka CLI tool as expected:
./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-001If you cannot reproduce it in a KoP standalone as well, you should check your configs first. And you can enable the debug level logs in KoP and upload the log file.
sumeet-zuora
Demogorgon314
jiangtao7
tuteng
Describe the bug When I enabled Kop in pulsar and setup a Kafka consumer (python) to consume from pulsar, it was failed with error.
To Reproduce Pulsar & Kop: 2.9.2.12
or Run the this code: ` from confluent_kafka import Consumer
Error from the consumer:
Expected behavior The consumer should work without error.
Additional context
./bin/kafka-console-consumer.sh --bootstrap-server pulsar-broker.pulsar-poc.svc.cluster.local:9092 --topic test-001.This issue, so far, is only happened on consumers which are coded with Kafka python library.