search

Nordstrom / kafka-connect-sqs

The SQS connector plugin provides the ability to use AWS SQS queues as both a source (from an SQS queue into a Kafka topic) or sink (out of a Kafka topic into an SQS queue).
Apache License 2.0
68 stars 40 forks source link

Usage of this library in Amazon Managed Streaming for Apache Kafka (MSK) and MSK Connect #50

Open weaje01 opened 4 months ago

weaje01 commented 4 months ago

We are trying to use the SQS connector with SQS as a source and AWS MSK as the destination using MSK Connect (https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect.html)

MSK Connect explicitly says: MSK Connect uses Kafka Connect 2.7.1, an open-source framework for connecting Apache Kafka clusters with external systems such as databases, search indexes, and file systems

Reviewing the latest versions of this library, they state: Running the connector on versions of Kafka Connect prior to 3.0 is not recommended.

We haven't been successful in getting the connector running and are debugging, but if I'm reading this correct, should I assume that this wouldn't be a recommended configuration? That this library isn't really suitable for sending messages to AWS MSK using MSK Connect?

dylanmei commented 4 months ago

While I've not run this connector with MSK Connect, I've run others that are built with KC 3.x so I don't expect a related issue.