mhowlett
commented
6 years ago Asynchonous Configuration
- the key parameter for tuning latency / throughput is queue.buffering.max.ms
- request.required.acks == 1 is appropriate.
- you can set acks == 0 and still get delivery reports but you'll only get local error information. this is typically not the ideal setting in most scenarios (acks == 1 is usually the best tradeoff).
- you might want to consider setting delivery.report.only.error to true.
- the rd_kafka_produce_batch function is simply a way of putting more than one message onto the internal librdkafka message queues at a time. It does not impact how messages are delivered to kafka. It is mostly useful when making a C function call is expensive (for example from a language binding).
Synchronous Configuration
- if you have high durability requirements, you might consider setting request.required.acks to -1 (all). However, this adversely affects latency, so 1 might be best for your scenario.
- for synchronous producing, set max.in.flight to 1 and wait for a DR before producing again.
- typically, you might also set message.send.max.retries to a high number (100000 or something effectively infinite) and let librdkafka handle all the retry logic.
- librdkafka can't do transactional batch delivery - there will be an individual DR per message.
- we hope to add idempotent producer and transaction capability relatively soon.
- you should set queue.buffering.max.ms to 0 when producing synchronously to minimize latency. batch.num.messages is not important in this scenario.
karthikeyanajendran1984
edenhill
We have to produce or publish a message to Kafka Server on synchronous and asynchronous mode in our producer application. we have either high volume of message publishing or lower volume OLTP type messages, also Scenario 1 : our application should return the acknowledgement to caller for each message or batched message(synchronous) Scenario 2 : our application can just log delivery acknowledgement in log whenever acknowledgements received(asynchronous).
We have an approach to instance can support either synchronously or asynchronous mode based on configuration. KAFKA_COMMUNICATION_TYPE : Asynchronous(Configuration) Approach to asynchronous communication(Both single & batch message publishing):
1) Two key parameters queue.buffering.max.ms, batch.num.messages to be consider and set for latency and throughput. 2) Register a delivery report callback functions to get acknowledgement status. 3) set request.required.acks to 1 for asynchronous non blocking call 4) Also librdkafka C API support two async calls (rd_kafka_produce and rd_kafka_produce_batch) to be used for single message processing and batched request. 5) invoke rd_kafka_poll at to serve the delivery report. 6) Once delivery report callback function get invoked by librdkafka, log the acknowledgement status message into producer application log file. Because here the caller who send the actual message, he will not wait for any response. So we log into log file and error status can be handled manually. Questions:
Do we need to set request.required.acks to 1 for asynchronous non blocking call ? Can you please comment on above asynchronous approach ?
KAFKA_COMMUNICATION_TYPE : Synchronous (Configuration) Approach to synchronous communication(Both single & batch message publishing):
1) Register a delivery report callback functions to get acknowledgement status. 2) set request.required.acks to 1 for asynchronous non blocking call 3) librdkafka C API support only on async calls and to support sync producer( already mentioned on https://github.com/edenhill/librdkafka/wiki/Sync-producer) 4) Here for single message processing, it has to be wait for acknowledgements for each messages in a loop based as mentioned on above link. 5) Once received the acknowledgement status from delivery report callback, send back the acknowledgment to caller. Because here the caller who send the actual message, he will wait for a response for each message. So we send acknowledgment status to caller. Questions:
1) As we have to support batch message processing in synchronous message, we have to send one acknowledgment status for batch message to caller. But there is no approach to be use in Sync Producer link (mentioned above) to use batch API(rd_kafka_produce_batch()), it has only used rd_kafka_produce(). Can you please give us whether it can be implementable or not advisable ? 2) As i look into batch API (rd_kafka_produce_batch), delivery report calls for each message inside the batch. For Ex : If we have publish 1000 message at one batch call, we will receive 1000 deivery reports through callback for each message. Please confirm ? 3) Do we also need to consider the key parameters(queue.buffering.max.ms and batch.num.messages) while performing synchronous call or only set request.required.acks is enough ?
COMMON: Please have a look and advise on the above approach.
Thanks in advance.
Regards, Karthikeyan Rajendran.