search

IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.57k stars 1.76k forks source link

Differentiate between client and server version of ErrMessageSizeTooLarge #543

Closed wvanbergen closed 1 year ago

wvanbergen commented 9 years ago

Right now, the Kafka broker can return ErrMessageSizeTooLarge if a message is deemed to large based on the broker configuration, but we also return it client-side in the producer based on the sarama configuration: https://github.com/Shopify/sarama/blob/53f2a54a5cfd662728648b20eb26e4a8e19b4887/async_producer.go#L255

For debugging purposes, it would be nice to be able to differentiate between the client and server-side version.

cc @andremedeiros @kvs @eapache

eapache commented 9 years ago

There are a lot of cases where we return kafka server errors directly:

At the time it seemed unnecessary to define a new local error when the kafka-provide value had the exact right meaning, but I guess this could lead to confusion...

wvanbergen commented 9 years ago

This case is a bit more confusing because it is determined by two configuration settings that live in different places that could be different from each other. This made debugging this error a lot harder.

Many of the other cases where we do this are not really influenced by different configuration settings, and we return what the broker would return anyway based on our best locally cached knowledge. I am not against a generic solution, but we jumped in the rabbit hole of adding context to these errors before without much luck.

Note: we also have ErrMessageTooLarge: https://github.com/Shopify/sarama/blob/master/errors.go#L37-L38

eapache commented 9 years ago

Ah you're right, that's legitimately confusing. Especially when you throw the consumer version of the error into the mix.

AusIV commented 4 years ago

I don't know that I have much more to add than a "me too", but I lost some critical time because this error is outright misleading. The error message:

kafka server: Message was too large, server rejected it to avoid allocation error.

It didn't even occur to me to look at the producer limits when the message specifically says the error was rejected by the server, and I knew the server limits were higher than the messages I was sending.

dnwe commented 1 year ago

Fixed by #2628 🎉