search

ibm-messaging / kafka-connect-mq-source

This repository contains a Kafka Connect source connector for copying data from IBM MQ into Apache Kafka.
Apache License 2.0
95 stars 83 forks source link

Guarantees when moving messages from MQ to Kafka #92

Closed mohanr closed 1 year ago

mohanr commented 2 years ago

I was looking for a detailed diagram showing protocols used to connect to MQ. All messages in MQ can move to a separate Queue so that the Kafka Connect Worker can pick it up.

  1. What protocol is used to pick messages from MQ ?
  2. Are there 'Exactly Once' guarantees ? Is it not guaranteed and we have to deduplicate and monitor properly ?
  3. The worker cluster will drop messages if Kafka itself has a problem.

In banking systems these guarantees are sought after as these messages are triggered by debit and credit activities.

mohanr commented 2 years ago

Are there benchmarks available ? It appears this integration isn't as simple as it seems. MQ buffering may be needed in case Kafka isn't able receive through the connector ? Some real use cases will also help.

jhughes24816 commented 1 year ago

Hi @mohanr, to answer your first set of questions:

  1. This connector uses a JMS client to read the messages from the MQ queue.
  2. At the moment there not an exactly once guarantee.
  3. This will depend on the settings used in your settings in your Kafka connect worker in the config of the Kafka Connect environment you can choose the acks value for all the connectors that are included in that worker. This is the same as the usual config for a producer apply to this config which you can find out more about here, by default the acks is all so Kafka Connect will know if a message fails to get to Kafka and will retry.

As for benchmarks, due to the variation of the circumstances of deployments for the connector it would be hard to provide any meaningful performance benchmarks for an individual connector. If Kafka can't receive the messages then, as mentioned in the answer to question 3, Kafka connect will retry.

Given the age of this issue I suggest we close this one out and if you have further questions you can raise a new issue and reference this conversation.