Sink: Frequent "Commit of offsets timed out" WARN level logs

[bmoe24x]

I have a Sink Connector that is getting an increasing number of logs warning Commit of offsets timed out such as:

[2020-04-22 17:51:07,383] WARN [<connector-name>|task-27] WorkerSinkTask{id=<connector-name>-27} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask:217)

This seems to start occurring only after processing has been going on for a while and when consuming a pretty large number of messages but once it starts it typically happens every minute in accordance with the default offset.flush.interval.ms

I read through some of the WorkerSinkTask and WorkerConfig code as well as the Connect Worker docs and found I could change offset.flush.timeout.ms or offset.flush.interval.ms but I am hoping someone might be able to point out why this could be happening and how to address it, rather than just increasing the timeout (unless that is actually a good fix?)

Doesn't seem to be an issue for tables I have idempotent UPSERTS on but for other tables that are append only I suspect this could increase duplicates.

Any thoughts? Any help is much appreciated

[Abdulsametileri]

Same issue, any update about this?

[ostap-palianytsia]

The same issue, are there any updates about this?

[kc93]

The same issue, are there any updates about this?

[regorvincentnono]

May I ask if somebody knows this issue?

[thiagohsp]

Same logs here. Any solutions?

[ruettere]

I also encounter the same issue, any idea how to solve this?

[ruettere]

I also encounter the same issue, any idea how to solve this?

I found the reason for this: in my case I didn't define a primary key in the target table where the upserts were done to, as soon as the primary key was added this WARN entries haven't appeared any more

[BrianSurrattDsg]

I also encounter the same issue, any idea how to solve this?

I found the reason for this: in my case I didn't define a primary key in the target table where the upserts were done to, as soon as the primary key was added this WARN entries haven't appeared any more

FYI, I've seen this error on a source connector which is upserting into multiple tables that do have a primary key. Does not seem to be broadly applicable.

[jslusher]

In my case, the only way I've found I can make these warnings go away is to reduce batch.size to a point where they stop. I'm under the impression that these warnings are because the connector is attempting to insert too many records for the database to handle, the transaction takes longer than offset.flush.timeout.ms, and offsets fail to be committed. I've been trying to get a better understanding of this problem because at the current batch rate I have to set to avoid these commit timeouts, my sinks are taking an unacceptable amount of time. It would be great if someone from confluent could chime in here.

[ruettere]

In my case, the only way I've found I can make these warnings go away is to reduce batch.size to a point where they stop. I'm under the impression that these warnings are because the connector is attempting to insert too many records for the database to handle, the transaction takes longer than offset.flush.timeout.ms, and offsets fail to be committed. I've been trying to get a better understanding of this problem because at the current batch rate I have to set to avoid these commit timeouts, my sinks are taking an unacceptable amount of time. It would be great if someone from confluent could chime in here.

I absolutely agree with you. I've been facing the same issue again and been working at this in the last days without finding an ideal configuration. I've tried several settings with max.poll.records, max.poll.interval.ms, enable.auto.commit, auto.commit.interval.ms, fetch.max.wait.ms, but nothing really worked. Having the log level set to TRACE, I couldn't get any more insights either. This week, I am gonna try Debezium Sink Connector. Excited about the results...

[mfvitale]

Which database are you using?

[ruettere]

We are using Oracle 19c, currently non CDB yet

[mfvitale]

In my case, the only way I've found I can make these warnings go away is to reduce batch.size to a point where they stop. I'm under the impression that these warnings are because the connector is attempting to insert too many records for the database to handle, the transaction takes longer than offset.flush.timeout.ms, and offsets fail to be committed. I've been trying to get a better understanding of this problem because at the current batch rate I have to set to avoid these commit timeouts, my sinks are taking an unacceptable amount of time. It would be great if someone from confluent could chime in here.

@jslusher Has your destination table some indices? If yes, try do delete it and re-add after the import.