search

Nordstrom / kafka-connect-sqs

The SQS connector plugin provides the ability to use AWS SQS queues as both a source (from an SQS queue into a Kafka topic) or sink (out of a Kafka topic into an SQS queue).
Apache License 2.0
68 stars 40 forks source link

Missing SQS_MESSAGE_ATTRIBUTE_PARTITION_KEY configuration in config def - SQS Source connector #47

Open aksinghgrd opened 3 months ago

aksinghgrd commented 3 months ago

org.apache.kafka.common.config.ConfigException: Unknown configuration 'sqs.message.attributes.partition.key' at org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:162) at org.apache.kafka.common.config.AbstractConfig.getString(AbstractConfig.java:197) at com.nordstrom.kafka.connect.sqs.SqsConnectorConfig.<init>(SqsConnectorConfig.java:33) at com.nordstrom.kafka.connect.sqs.SqsSourceConnectorConfig.<init>(SqsSourceConnectorConfig.java:51) at com.nordstrom.kafka.connect.sqs.SqsSourceConnectorTask.start(SqsSourceConnectorTask.java:61) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.initializeAndStart(AbstractWorkerSourceTask.java:277) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:227)

Missing configuration for SQS_MESSAGE_ATTRIBUTE_PARTITION_KEY in class SqsSourceConnectorConfig while creating ConfigDef CONFIG_DEF

davidkurutz-reddit commented 3 months ago

Upvoting this -- same issue here -- code looks like sqs.message.attribute.partition.key was implemented in SinkConnector and in Config objects -- we are trying to use the source connector only and getting this error.

org.apache.kafka.common.config.ConfigException: Unknown configuration 'sqs.message.attributes.partition.key'
    at org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:162)
    at org.apache.kafka.common.config.AbstractConfig.getString(AbstractConfig.java:197)
    at com.nordstrom.kafka.connect.sqs.SqsConnectorConfig.<init>(SqsConnectorConfig.java:33)
    at com.nordstrom.kafka.connect.sqs.SqsSourceConnectorConfig.<init>(SqsSourceConnectorConfig.java:51)
    at com.nordstrom.kafka.connect.sqs.SqsSourceConnectorTask.start(SqsSourceConnectorTask.java:61)
dylanmei commented 2 months ago

Let us know whether release v1.6.1 resolves the issue.

rejthy commented 2 months ago

It does not work, because there is no config definition for key 'sqs.message.attributes.partition.key' in class SqsSourceConnectorConfig.java

dylanmei commented 4 weeks ago

You should find that sqs.message.attributes.partition.key is truly optional in release v1.6.2.