search

ibm-messaging / kafka-connect-mq-sink

This repository contains a Kafka Connect sink connector for copying data from Apache Kafka into IBM MQ.
Apache License 2.0
35 stars 43 forks source link

Question > Recommended low-latency configuration #40

Closed Musikolo closed 3 years ago

Musikolo commented 3 years ago

Hi,

I'm using the IBM MQ Sink connector successfully for a while. Currently, I'm pulling records from several topics and pushing those to a single MQ queue. I have some related records of different topics that are produced within a few seconds difference. I noticed that some times, the connector buffers a bunch of records together and push them all at the same time to the target MQ queue. The destination app is processing the records of MQ queue in a multi-threading fashion. The problem I'm facing is that because some records and time-dependent and they are available for processing almost at the same time, this is causing sporadic race-condition issues.

So, in order to solve (or at least greatly mitigate) this issue, I'm trying to find out if there is a configuration optimizes the connector from latency respective, i.e., that pushes records to MQ as fast as possible. Because we don't have a lot of traffic, I think it should scale well. Currently, I was thinking of using the following configuration:

confluent.topic.consumer.fetch.min.bytes=1

However, I don't know if this will cause the desired behavior and whether there is a better way to achieve our goal.

Thank you!

AndrewJSchofield commented 3 years ago

Hi, a couple of suggestions. You could try setting offset.flush.interval.ms to a lower number which has the side-effect of reducing the duration of the batches. You could also try setting a configuration for the built-in Kafka consumer that the Kafka Connect framework is using internally.

Hope this helps.

ryancrawcour commented 3 years ago

@AndrewJSchofield is offset.flush.interval.ms a connector specific configuration, or a Kafka Connect configuration?

AndrewJSchofield commented 3 years ago

Kafka Connect configuration. The batching is controlled by Kafka Connect.

Musikolo commented 3 years ago

Hi @AndrewJSchofield ,

Thanks for your prompt response. Looking at the suggested option at the Confluent documentation site I've found the following:

image

After reading the above descriptions, I'm not sure if those 2 properties are dependent on each other, or at least, whether it is expected offset.flush.interval.ms > offset.flush.timeout.ms. So, considering the former default value is 60 seconds and the latter 5 seconds, I'm not sure if using something like offset.flush.interval.ms=1000 & offset.flush.timeout.ms=500 would cause some records to be cancelled.

Thoughts?

AndrewJSchofield commented 3 years ago

I think it is important that offset.flush.interval.ms > offset.flush.timeout.ms. I think you could reduce to these to a few seconds, but that's not really very low latency like you suggested.

AndrewJSchofield commented 3 years ago

Closing dormant issue.