Closed d-t-w closed 7 months ago
@d-t-w thank you for the detailed report.
SdkClientException: Multiple HTTP implementations were found on the classpath.
This is expected. When you have multiple HTTP clients in your classpath, the SDK doesn't know which one to use. You need to tell the SDK which one to use by (1) setting the software.amazon.awssdk.http.service.impl
system property; or (2) setting the http client programmatically in the service client builder:
S3Client s3Client = S3Client.builder()
.httpClientBuilder(ApacheHttpClient.builder())
.build();
(System/setProperty "software.amazon.awssdk.http.service.impl" "software.amazon.awssdk.http.apache.ApacheSdkHttpService")
It is not clear how this setting impacts any of the libraries, but Glue/LM/MSK appear to work with that setting and IRSA roles are resumed.
The Java SDK will honor this setting across all the service clients vended by the SDK. But note that the software.amazon.glue/schema-registry-serde
library is not maintained by we the Java SDK team, so I can't say how setting the "software.amazon.awssdk.http.service.impl" system property to use Apache Client will impact the rest of the library that expected to use UrlConnection Client.
Let me know if this makes sense.
No worries @debora-ito, but I think this issue deserves a bit more thought.
This is expected. When you have multiple HTTP clients in your classpath, the SDK doesn't know which one to use.
As a user of the AWS SDK I do not expect that adding a library will cause IRSA to stop working.
You need to tell the SDK which one to use by (1) setting the software.amazon.awssdk.http.service.impl system property; or (2) setting the http client programmatically in the service client builder:
The software.amazon.glue/schema-registry-serde
library sets the UrlConnectionHttpClient programatically.
The library does not provide any way for me to configure a different http client.
The software.amazon.awssdk/auth WebIdentityTokenCredentialsProvider
class uses a serviceLoader for http impl.
Presumably setting software.amazon.awssdk.http.service.impl
only applies as a default when using the serviceLoader technique? To be clear I have no idea if that is true or if setting that variable overrides any programatic settings.
I assume setting software.amazon.awssdk.http.service.impl
impacts WebIdentityTokenCredentialsProvider behaviour.
Multiple factors lead this to being a broad issue:
[1] This issue is easily triggered
The WebIdentityTokenCredentialsProvider created by the DefaultCredentialsProvider will fail if there are multiple HTTP Client implementations on the classpath.
[2] This issue is likely to be triggered
Adding any library that causes multiple http implementation on the classpath will trigger this issue. E.g. software.amazon.glue/schema-registry-serde
, or (randomly picking one..) org.apache.iceberg.iceberg/iceberg-aws
.
[3] This issue is consequential
This issue causes IRSA to stop working.
Our product runs on EKS and requires IRSA to call the AWS Licence Manager as a part of AWS Marketplace integration. Adding the scema-registry-serde library causes that integration to fail.
[4] This issue is hard to identify
Setting up EKS with IRSA is a multi-step process with lots of moving parts. When it doesn't work you tend to assume that you have set your cluster up incorrectly, missed some configuration, etc. The IRSA troubleshooting pages suggest confirm ingOIDC configuration, IAM role trust policies, etc.
The thought that a dependency conflict within your JAR file has caused the failure isn't one that springs to mind.
[5] This issue is hard to debug
The AwsCredentialsProviderChain
class logs a debug log-line, that is very low visibility.
Finding the root cause of this issue required debug monitoring of an application in a full EKS+IRSA environment.
[6] This issue is encountered
The AWS iam-auth and serdes libraries have these open tickets:
https://github.com/awslabs/aws-glue-schema-registry/issues/151 https://github.com/awslabs/aws-glue-schema-registry/issues/157 (specifically the comments at the bottom) https://github.com/aws/aws-msk-iam-auth/issues/55
[7] This issue probably more broadly applicable
My examples are from Kafka related libraries, but basically anything that introduces a second http impl will break IRSA.
[8] The workaround is imprecise
How can I know what the impact of setting software.amazon.awssdk.http.service.impl
is across the different libraries in my application? Should I use UrlConnection / ApacheClient / NettyClient, etc?
Suggestions
Someone with a better knowledge of the libraries than me could consider the impact of:
Thanks! Derek.
Just spent way too much time investigating an issue related to this as well. Raising visibility of the log line, particularly in the context of IRSA, sounds like a MUST to me as well.
Any update about this issue? I spent too much time of debugging because of this issue too.
@d-t-w @maximethebault @Min3953
We changed the rule when multiple implementations are found in the classpath. Now, instead of throwing an error, the SDK will choose one http client based on priority. The priority order is defined in the ClasspathSdkHttpServiceProvider. The change was released in SDK version 2.22.0.
So in this case, WebIdentityTokenCredentialsProvider will not fail anymore, granting version 2.22.0
or greater is being used. This solves the root cause of this issue. Let us know what you think.
Hi @debora-ito that sounds like a good solution to me, thanks very much!
@d-t-w Good to know, thank you for the follow-up. Resolving this.
This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.
Describe the bug
Kpow for Apache Kafka is an enterprise toolkit for Apache Kafka that includes multiple AWS libraries including LicenseManager, MSK Iam Auth, and AWS Glue.
Including the AWS Glue dependency silently breaks IRSA, causing the pod to run under the NodeInstanceRole rather than the properly configured IRSA role.
This is likely to impact projects intending to use IRSA with MSK, as the MSK IAM library and AWS Glue libraries are very likely to be included in those projects. See: https://github.com/aws/aws-msk-iam-auth/issues/55
Isolating this error required a full deploy into an IRSA enabled EKS environment with debug logging in place.
Expected Behavior
Kpow prior to the addition of the AWS Glue dependency implemented IRSA correctly.
We expect to add the AWS Glue dependency to Kpow without breaking IRSA.
Current Behavior
After adding the AWS Glue dependency Kpow reverted to operating under the EKS Node Instance role:
Turning on debug logging shows the WebIdentityTokenCredentialsProvider is not executing due to an error resolving the http implementation,
The error is caused by conflicting http implementations brought in by:
LicenseManager or MSK (Iam Auth): ApacheHttpClient AWS Glue: URLConnection
Reproduction Steps
See minimum viable reproducer here: https://github.com/factorhouse/aws-irsa-deps-reproducer
Possible Solution
Manually set the http client implementation:
It is not clear how this setting impacts any of the libraries, but Glue/LM/MSK appear to work with that setting and IRSA roles are resumed.
Additional Information/Context
No response
AWS Java SDK version used
2.18.20
JDK version used
java --version openjdk 11.0.16 2022-07-19
Operating System and version
Mac OS