Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage
Apache License 2.0
95 stars 20 forks source link

error in consumer-perf-test #479

Closed freshtang closed 10 months ago

freshtang commented 10 months ago

What happened?

1.create tieredTopic bin/kafka-producer-perf-test.sh --topic tieredTopic0105 --num-records=100000000 --throughput 100000 --record-size 1000 --producer-props acks=1 batch.size=16384 bootstrap.servers=localhost:9092

2.produce messages with kafka-producer-perf-test bin/kafka-producer-perf-test.sh --topic test --num-records=100000000 --throughput 100000 --record-size 1000 --producer-props acks=1 batch.size=16384 bootstrap.servers=localhost:9092

3.consume with kafka-consumer-perf-test.sh bin/kafka-consumer-perf-test.sh --topic tieredTopic0105 --bootstrap-server localhost:9092 --messages 100000 --group test

image

image

What else do we need to know?

broker uses aliyun oss as s3

jeqo commented 9 months ago

@freshtang just as reference, this is caused because of how timeouts are handled on the broker side, see #483 for future resolution.

As a workaround, consumers can increase their fetch.max.wait.ms to e.g. 2sec