search

ibm-messaging / kafka-connect-mq-sink

This repository contains a Kafka Connect sink connector for copying data from Apache Kafka into IBM MQ.
Apache License 2.0
35 stars 43 forks source link

MQ Manager ( Ver: 7.5.0.6) not receiving JMSCorrelationID #71

Open jobykurianiag opened 6 days ago

jobykurianiag commented 6 days ago

Hello,

We are using "Kafka connect mq sink" for sending message from AWS MSK Topic to MQ Manager (WebSphere MQ Ver: 7.5.0.6). The connector deployed in ECS Fargate. We are using below config to send message, unfortunately MQ Manager not receiving "JMSCorrelationID".

Kafka - AWS MSK Connector version - v1.5.2 Task - AWS ECS Service

mq-sink.json

{ "name": "name-sink", "config": { "connector.class": "com.ibm.eventstreams.connect.mqsink.MQSinkConnector", "tasks.max": "1", "topics": "topic name",
"key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.storage.StringConverter", "mq.queue.manager": "XXXX", "mq.connection.name.list": "XXXX", "mq.channel.name": "XXXXX", "mq.queue": "XXXXX", "mq.message.body.jms": "false", "mq.message.builder": "com.ibm.eventstreams.connect.mqsink.builders.DefaultMessageBuilder", "mq.message.builder.key.header": "JMSCorrelationID", "offset.storage.file.filename": "", "mq.userid":"XXXXX" } }

But when we sent same message to another MQ Manager with latest version (IBM MQ v9.2), it is working for them (they are receiving "JMSCorrelationID" as part of message).

  1. Is there any minimum version requirement for MQ Manager, in order to use mq sink connector ?
  2. If MQ Version is not supported, is there any alternative option to send "JMSCorrelationID" ?
dalelane commented 6 days ago

Did you mean to set JMSCorrelationID and to set JMS to false? At first glance, that seems like an unusual combination.

cf. https://github.com/ibm-messaging/kafka-connect-mq-sink/blob/main/README.md#key-support-and-partitioning

... Because these values are set using JMS message properties, they only have an effect if mq.message.body.jms=true is set ...

mamwl1 commented 6 days ago

Dale,

All MQ messages can have a correlation ID regardless of whether they are sent in "JMS format" or not. The only difference is that non-JMS correlation IDs can only be 24 bytes long as they have to fit in the MQMD. JMS format ones can be bigger if needed.

The link you posted about needing mq.message.body.jms=true to be set is I think referring to the topic/partition and offset which are message properties and therefore do need JMS format messages.