search

tulios / kafkajs

A modern Apache Kafka client for node.js
https://kafka.js.org
MIT License
3.75k stars 527 forks source link

Unnecessarily long delay after CONCURRENT_TRANSACTIONS #304

Open Nevon opened 5 years ago

Nevon commented 5 years ago

See https://issues.apache.org/jira/browse/KAFKA-5477

296 introduced a retry when receiving CONCURRENT_TRANSACTIONS error on idempotent producer initialization. The broker aborts the transaction, which means that we can just retry the operation. However, the default retry time is 300ms, which in a scenario where we are creating transactions back to back would introduce a significant latency (see the ticket I linked above for an explanation of why).

In Kafka 2, they might make commit block until the transaction is actually committed, which would solve this, but in the meantime we should be able to just override the retry time to a low value in case of CONCURRENT_TRANSACTIONS error.

rcbjBlueMars commented 4 years ago

Hello. I know this has been open for a while, but we started experiencing a situation similar to what you are describing after we upgraded to Kafka 2.5 (from Kafka 2.3). Before the upgrade, we didn't see the 300ms pause. Afterwards, we did. Would that be expected upgrading from Kafka 2.3 to Kafka 2.5?

We are using kafkajs 1.12.0.

Thanks,

RCBJ