kafka: support fetch.max.bytes broker configuration

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

https://redpanda.com

9.37k stars 575 forks source link

kafka: support fetch.max.bytes broker configuration #3697

Open dotnwat opened 2 years ago

dotnwat commented 2 years ago

KIP-541
See https://github.com/redpanda-data/redpanda/issues/3409 for more details as per @jcsp's comment.

JIRA Link: CORE-832

jcsp commented 2 years ago

This probably maps pretty directly to https://github.com/vectorizedio/redpanda/issues/3409 (see comment there about the two configs we currently have related to limiting fetch bytes)

LenaAn commented 2 years ago

initial plan of attack:

understand why fetch_max_bytes wasn't working
unify fetch_max_bytes and kafka_max_bytes_per_fetch
implement more general solution based on https://github.com/redpanda-data/redpanda/pull/3420

@jcsp does that sound reasonable?

jcsp commented 2 years ago

@LenaAn Exactly :ok_hand:

You may also find the new many_partitions_test.py a useful basis for testing this -- when it constructs a topic with 1024 partitions and then reads with a single consumer, that should stress the fetch size.

dotnwat commented 2 years ago

This probably maps pretty directly to https://github.com/redpanda-data/redpanda/issues/3409

lots of good discussion in #3409

LenaAn commented 2 years ago

As agreed with @piyushredpanda , this has lower priority than ci failures now