search

ibm-messaging / kafka-connect-mq-source

This repository contains a Kafka Connect source connector for copying data from IBM MQ into Apache Kafka.
Apache License 2.0
95 stars 83 forks source link

While using 6 tasks and connect worker gets stopped and started due to a wrong format message , Duplicate messages are received in Kafka #129

Closed AishD3 closed 7 months ago

AishD3 commented 9 months ago

What happened?

While using 6 tasks and connect worker gets stopped and started due to a wrong format message , Duplicate messages are received in Kafka

Steps To Reproduce

While using 6 tasks and connect worker gets stopped and started due to a wrong format message , Duplicate messages are received in Kafka

Environment

While using 6 tasks and connect worker gets stopped and started due to a wrong format message , Duplicate messages are received in Kafka

Anything else?

While using 6 tasks and connect worker gets stopped and started due to a wrong format message , Duplicate messages are received in Kafka

Version

1.3.4 (Default)

Relevant log output

While using 6 tasks and connect worker gets stopped and started due to a wrong format message , Duplicate messages are received in Kafkav

Code of Conduct

matuagarwal commented 9 months ago

Hi @AishD3

Exactly once delivery (which avoid duplication) is available only with MQ v2 connector https://ibm.github.io/event-automation/connectors/kc-source-ibm-mq2/installation

dalelane commented 7 months ago

Closing (as the connector here only offers at-least-once delivery, so duplicate messages are possible when the connector is restarted)

For use cases where exactly-once assurances are required, we recommend using the connector that Matu links to above.