message.max.bytes documentation

[coral-waters]

Someone in the community slack is asking about the message.max.bytes setting. It's not in the product doc at all, but is mentioned in this blog - https://redpanda.com/blog/fast-and-safe/

slack thread: https://redpandacommunity.slack.com/archives/C01AJDUT88N/p1645746186920799

JIRA Link: CORE-851

[adhasahar97]

Bump. Really need this.

[dotnwat]

thanks @adhasahar97. it looks like we added this recently with the configuration option kafka_batch_max_bytes

@mmaslankaprv did you recently add a max message size limitation in the kafka layer?

[mmaslankaprv]

Hello @adhasahar97, there are two broker configuration properties setting up the message and batch size limits:

kafka_batch_max_bytes - Maximum size of a batch processed by server. If batch is compressed the  limit applies to compressed batch size
kafka_request_max_bytes - Maximum size of a single request processed via Kafka API

There is also per topic configuration for max batch size:

max.message.bytes

[kargh]

@mmaslankaprv I assume that the broker configuration goes into redpanda.yaml. If so, which section?

I ran into this exact issue about a half hour ago when my mirroring blew up due to large message sizes. I changed the topic max.message.bytes but would like to set a cluster-wide max.

Thanks!

[mmaslankaprv]

Broker configuration can be changed with:

rpk cluster config set <key> <value>

[fabriziofortino]

I stumbled into

unable to produce record: MESSAGE_TOO_LARGE: The request included a message larger than the max message size the server will accept.

I spin redpanda up with the following options:

  redpanda:
    image: docker.redpanda.com/vectorized/redpanda:v22.3.4
    command:
      - redpanda start
      - --smp 1
      - --memory 512M
      - --overprovisioned
      - --node-id 0
      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
      - --set redpanda.kafka_batch_max_bytes=20971520
      - --set redpanda.kafka_request_max_bytes=20971520

The topic is created with rpk topic create my_topic -c max.message.bytes=20971520 but I am still not able to ingest messages larger than 1MB.

@mmaslankaprv am I doing anything wrong in there? Thanks

[daisukebe]

@fabriziofortino, if you're still seeing MESSAGE_TOO_LARGE, it's possible that the property isn't set in the server side. Can you let us know the result of rpk cluster config get kafka_batch_max_bytes?

[fabriziofortino]

@daisukebe I set rpk cluster config set kafka_batch_max_bytes 104857600 but I still see MESSAGE_TOO_LARGE. I send messages with jq -c . large.json | rpk topic produce my_topic I wonder if the issue is in the producer.

[daisukebe]

@fabriziofortino, seems like it's bounded by the underlying go library, franz-go, which enforces the max batch size at the client side. Apparently there's no way to tweak from rpk side, correct me if I'm wrong @twmb, but I think you can test with the bench application bundled in franz-go.

$ git clone git@github.com:twmb/franz-go.git
$ cd franz-go/examples/bench

You can tweak the client side limitation via -batch-max-bytes. Given the cluster defaults to 1MB, this will fail

$ go run . -topic test  -brokers localhost:9092 -batch-max-bytes 1058576
produce error: MESSAGE_TOO_LARGE: The request included a message larger than the max message size the server will accept.
exit status 1

Change the server side limitation to 100MB and give it a try. It should work.

$ go run . -topic test  -brokers localhost:9092 -batch-max-bytes 1058576
254.79 MiB/s; 2671.68k records/s

Does this help?

[twmb]

https://github.com/redpanda-data/redpanda/issues/7641

[samzong]

rpk --brokers <host> topic alter-config <topic_name> --set max.message.bytes=4294897000

Using this command can directly expand max.message.bytes

[JRobTS]

I had the same issue as @fabriziofortino

For me, adding --max-message-bytes to my rpk topic produce solved the issue.

There's a red herring in the error message: unable to produce record: MESSAGE_TOO_LARGE: The request included a message larger than the max message size the server will accept.

It should probably read as something like this instead: unable to produce record: MESSAGE_TOO_LARGE: The request included a message larger than the max message size the producer will send.

... or better yet... unable to produce record: MESSAGE_TOO_LARGE: The request included a message larger than the max message size the producer will send (1 MiB). This can be increased using --max-message-bytes

Ticket created for this here: #12389