search

confluentinc / kafka-connect-jdbc

Kafka Connect connector for JDBC-compatible databases
Other
19 stars 955 forks source link

MongoDB Source Connector - configuration validation runs before replacement when using a Config Provider #1319

Open lincoln42 opened 1 year ago

lincoln42 commented 1 year ago

Environment: Confluent Stack running using Docker Compose (Kafka, KafkaConnect, Zookeeper, etc.) Version: 7.3.0 ConfigProvider being used: SecretsManagerConfigProvider - https://github.com/jcustenborder/kafka-config-provider-aws Mongo Source Connector being used: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb

Problem: When using the Kafka Connect REST API to either validate or create a new MongoSourceConnector instance when using the above configuration provider to replace the values of secret configuration items in the configuration as below. I get the error - Invalid value "${secretmanager:dev/db/mongo/test/connection:uri}" for configuration connection.uri

Additionally, I am using an AWS Profile to provide the AWS credentials and I can confirm that I can fetch the secrets from AWS Secrets Manager just fine via the AWS CLI for the profile being used. The AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE environment variables have been set on the Kafka Connect container as well. Also, the Mongo Source connector works just fine when not using a config provider and with a directly supplied connection.url value.

"connector.class" : "com.mongodb.kafka.connect.MongoSourceConnector",
"config.providers" : "secretmanager",
"config.providers.secretmanager.class"  : "com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider",
"config.providers.secretmanager.param.aws.region" : "us-east-1",
"tasks.max" : 1,
"key.converter" : "io.confluent.connect.avro.AvroConverter",
"value.converter" : "io.confluent.connect.avro.AvroConverter",
"errors.tolerance" : "all",
"errors.log.enable": "true",
"errors.log.include.messages" : "false",
"errors.deadletterqueue.context.headers.enable": "true",
"errors.deadletterqueue.topic.name" : "dlq",
"connection.uri" : "${secretmanager:dev/db/mongo/test/connection:uri}",

Could this issue be linked to, or a regression of, https://github.com/confluentinc/kafka-connect-jdbc/issues/737 ?

OneCricketeer commented 1 year ago

Hello. This repo is for Confluent's JDBC Source/Sink. MongoDB's Kafka Connector is a different repo.