sgrabbit
commented
4 months ago Hi. Did you find any fix for this?
Closed michal-rybinski closed 2 months ago
sgrabbit
commented
4 months ago Hi. Did you find any fix for this?
michal-rybinski
commented
4 months ago nope, but was not looking too hard really
ilyas1uphealth
commented
3 months ago The issue appears to be with the version of the sdk used in this project -- v1 is not supported https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html https://github.com/aws/aws-sdk-java/issues/3062#issuecomment-1861905542
I'm not very familiar with this space, but based on this commit it looks like the project uses v1.12.638 on the latest release https://github.com/aws/aws-msk-iam-auth/commit/ee55c0ffdf98f6cef45f25ee5179155ca9515d77
It looks like to support PodIdentity, this project would need to update that 2.21.30 however I have no idea what that entails.
sidyag
commented
3 months ago Can you try this with the PR: https://github.com/aws/aws-msk-iam-auth/pull/157
This updates all the internal usages of Java sdk v1 to v2.
rajarshp
commented
3 months ago Hello,
can this be fixed and release asap ? this is impacting our implementation and we are stuck because of this issue and we are eagerly looking for a solution on this
michal-rybinski
commented
3 months ago Can you try this with the PR: #157
This updates all the internal usages of Java sdk v1 to v2.
Hi @sidyag , sorry for long radio silence,
I'll try to see if I can test it soon, although have not done it before without a release I can point my terraform automation at, so will have to research how to do it.
michal-rybinski
commented
2 months ago I've noticed that it was merged and see there is version 2.0.3 available now with "Upgrade AWS SKD version" comment (probably should be SDK?) so I gave it a spin today but unfortunately still see the error:
WARN Exception loading credentials. Retry Attempts: 0 (software.amazon.msk.auth.iam.internals.MSKCredentialProvider)
aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain@17ead63f: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@f94bdf7: Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), aws_msk_iam_auth_shadow.com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@6815eca9: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host should resolve to a loopback address or have the full URI be HTTPS.]]
at aws_msk_iam_auth_shadow.com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:142)
at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:158)
at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.getCredentials(MSKCredentialProvider.java:145)
at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:100)
at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139)
at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:534)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:433)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:332)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:273)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1381)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312)
at java.base/java.lang.Thread.run(Thread.java:829)although I am definitely using the new library
# ls ../libs/aws*
../libs/aws-msk-iam-auth-2.0.3-all.jarDo I need to change client.properties in any way (or something else) to use it?
michal-rybinski
commented
2 months ago although.... I just noticed that the 2.0.3 release date was 17th of Jan.... :/
michal-rybinski
commented
2 months ago ok, got gradle set up and tried to build the newest version cloned but regardless of version used (8.7 or 7.6.4) it fails:
8.7
# gradle clean build
Starting a Gradle Daemon (subsequent builds will be faster)
FAILURE: Build failed with an exception.
* What went wrong:
org/gradle/api/plugins/MavenPlugin
> org.gradle.api.plugins.MavenPlugin
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.7/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 19s7.6.4
# gradle clean build
Welcome to Gradle 7.6.4!
Here are the highlights of this release:
- Added support for Java 19.
- Introduced `--rerun` flag for individual task rerun.
- Improved dependency block for test suites to be strongly typed.
- Added a pluggable system for Java toolchains provisioning.
For more details see https://docs.gradle.org/7.6.4/release-notes.html
Starting a Gradle Daemon (subsequent builds will be faster)
> Task :compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :javadoc FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':javadoc'.
> Javadoc generation failed. Generated Javadoc options file (useful for troubleshooting): '/aws-msk-iam-auth/build/tmp/javadoc/javadoc.options'
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.6.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 36s
Not sure about build fail, as it is passing for me. I am working on running tests, and release.
michal-rybinski
commented
2 months ago ok, got it working (installing java-devel helps....)
and the jar generated with gradle clean shadowJar works fine now! :1st_place_medal:
rajarshp
commented
2 months ago Hello, when can we expect this fix to be released?
sidyag
commented
2 months ago The PRs have been released as part of 2.1.0 Could you try and see if that resolves the issue.
michal-rybinski
commented
2 months ago Yes, I can confirm it works fine with 2.1.0!
Thank you, closing :)
github-actions[bot]
commented
2 months ago This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.
Is this mechanism also compatible with passing credentials through Pod Identity?
I am testing switching from IRSA to Pod Identity in a K8s cluster and pods which have IRSA configured on them can successfully connect to MSK using this module, whereas each time I try to do it via pod with Pod Identity role, it fails, although I can connect to other services allowed by the IAM role so the mechanism itself works fine.
Is there some specific configuration required for this to work here?
I am a bit at a loss if this is this library problem or kafka's client, but I tried using the newest 3.6.1 from here https://archive.apache.org/dist/kafka/3.6.1/ alongside with version 2.0.2 of this library and it still throws same errors:
AWS Docs are saying here https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html:
"EKS Pod Identities have been added to the Container credential provider which is searched in a step in the default credential chain. If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an EKS Pod Identity association for the same workload."
I am wondering if the problem is in:
Would appreciate any hints here.
Thanks