kafka-connect-s3 - Trusted Account Credentials Documentation

[Tamir-Schwarz]

Hi,

We are using kafka-connect-s3 connector version 5.5.1. We tried to use the Trusted Account Credentials feature.

According to the documentation, we need to add the following properties:

s3.credentials.provider.class=AwsAssumeRoleCredentialsProvider sts.role.arn=arn:aws:iam::012345678901:role/my-restricted-role sts.role.session.name=session-name sts.role.external.id=external-id

It didn't work for us.

After digging the code, we found that in order to make it work we need to add a prefix 's3.credentials.provider' to 'sts.role.arn', 'sts.role.session.name' and 'sts.role.external.id'

Following is the configuration that worked for us:

s3.credentials.provider.class=AwsAssumeRoleCredentialsProvider s3.credentials.provider.sts.role.arn=arn:aws:iam::012345678901:role/my-restricted-role s3.credentials.provider.sts.role.session.name=session-name s3.credentials.provider.sts.role.external.id=external-id

In addition, It worth mentioning that 'external.id' is mandatory, while it is not mandatory on AWS

[hafizmujadidKhalid]

@Tamir-Schwarz How it worked for you? I am getting following error:

connector.class=io.confluent.connect.s3.S3SinkConnector
s3.region=eu-central-1
tasks.max=2
topics=kafka-demo
s3.credentials.provider.class=io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider
s3.credentials.provider.sts.role.arn=arn:aws:iam::xxxxxxxx:role/-demo-owner-readwrite-role
s3.credentials.provider.sts.role.session.name=session-name
s3.credentials.provider.sts.role.external.id=data-key

[Worker-0db15ad1ff259517f] Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

[timvryan]

@hafizmujadidKhalid Getting the same error, did you find a way to solve?

[hafizmujadidKhalid]

@hafizmujadidKhalid Getting the same error, did you find a way to solve?

No, it did not work for me.

[akhon]

same problem

[mjallday]

https://github.com/confluentinc/kafka-connect-storage-cloud/blob/c633f0828d591ef704f37ab3c4f2a4d1ef447d5f/kafka-connect-s3/src/main/java/io/confluent/connect/s3/S3SinkConnectorConfig.java#L806-L837 https://github.com/confluentinc/kafka-connect-storage-cloud/blob/c633f0828d591ef704f37ab3c4f2a4d1ef447d5f/kafka-connect-s3/src/main/java/io/confluent/connect/s3/storage/S3Storage.java#L172-L176 https://github.com/confluentinc/kafka-connect-storage-cloud/blob/c633f0828d591ef704f37ab3c4f2a4d1ef447d5f/kafka-connect-s3/src/main/java/io/confluent/connect/s3/auth/AwsAssumeRoleCredentialsProvider.java#L86-L101

first glance it looks like the credential provider is being invoked correctly. we don't see similar issues when using a regular role so it's something related to the assume role configuration.

[gotpist1]

Im also getting the same error using localstack and root role. Any updates on this? Also, specifying AWS_REGION ENV variable gives another error:

org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@16971f4e: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@7cecef68: Failed to connect to service endpoint: ]
    at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:138)```

[karlamaria100]

Region issue still happening

[chriline]

https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html#credentials-providers is still using s3.credentials.provider.sts.role.arn, s3.credentials.provider.sts.role.session.name and s3.credentials.provider.sts.role.external.id while https://docs.confluent.io/kafka-connectors/s3-sink/current/configuration_options.html is using sts.role.arn, sts.role.session.name, and sts.role.external.id.

I believe we're also running into @hafizmujadidKhalid's issue: We've set up io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider like so

s3.credentials.provider.class=io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider
s3.credentials.provider.sts.role.arn=arn:aws:iam::xxxxxxxx:role/my-role
s3.credentials.provider.sts.role.session.name=session-name
s3.credentials.provider.sts.role.external.id=external-id

and get the following error

[Worker-0f3f4a9ad02a8f900] Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

[AyoubOukh]

getting the same issue