Closed starchx closed 4 years ago
endre-synnes
commented
5 years ago Having the same issue using the AWS Java SDK version: 1.11.653.
According to the documentation this version of the SDK should also work fine. My application uses the role of the node instead of the one I have added through a service account.
starchx
commented
5 years ago @endre-synnes The issue is due to WebIdentityTokenFileCredentialsProvider is not in the default credentials provider chain
The workaround for now is to specify the provider in the client initialization:
import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
...
S3Client s3 = S3Client.builder()
.credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
.region(region)
.build();Looking forward that WebIdentityTokenFileCredentialsProvider to be added to the default credentials provider chain.
endre-synnes
commented
5 years ago @starchx Thank you! That solved the issue😄 Hope that the WebIdentityTokenFileCredentialsProvider will be added as a default soon!
dagnir
commented
5 years ago Hi all, this should be part of the default chain. We will prepare a change for this.
thundergolfer
commented
5 years ago @endre-synnes I am also experiencing the same problem despite the documentation saying it should work. We should file an issue at https://github.com/aws/aws-sdk-java/issues
zoewangg
commented
4 years ago Fixed via #1501 and released as part of 2.10.11
thereisnospoon
commented
4 years ago Hi,
It still doesn't work for me with 2.10.57. The sdk uses node role instead of service account role.
UPDATE It works. I just had to add
securityContext:
fsGroup: 65534to the pod spec in order to make sure containers have access to the service token file.
imcheck
commented
4 years ago Hi, It still doesn't work for me with 2.10.72.
It seems that java sdk doesn't get credentials from AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN envs.
(FYI, I already added security context to pods to be able to access the token file)
thereisnospoon
commented
4 years ago @imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.
jiang-wei
commented
4 years ago @imcheck make sure you have
aws-java-sdk-stsdependency packaged for your application as well.
Not really.
When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path
So I searched and get StsWebIdentityCredentialsProviderFactory.java
Then I included
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sts</artifactId>
</dependency>It just works :)
gargkshitiz
commented
4 years ago Hi Jiang,
Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java
jiang-wei
commented
4 years ago Hi Jiang,
Could you please let me the code snippet you have used with
StsWebIdentityCredentialsProviderFactory.java
S3Client s3 = S3Client.builder()
.region(region)
.build();and include sts with the latest version from maven repo, today it is 2.11.14
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sts</artifactId>
</dependency>Thank you Jiang. This was conflicting with aws sdk jar I was using. Latest versions of sdk and sts helped me.
campbel
commented
4 years ago This is an interesting discussion and somewhat related to a use case I am trying to figure out. My service needs to access resources in a different AWS account from EKS, so I want to use profiles and assume role using a config like this:
[profile eks-account-role]
role_arn = <eks-account-role-arn>
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token
[profile other-account-role]
role_arn = <other-account-role-arn>
source_profile = eks-account-role
role_session_name = x-account-sessionUsing the following code to access the STS client (for other purposes than assuming roles).
stsClient = StsClient.builder().build();Then I set the environment variable AWS_PROFILE=other-account-role in the container.
But when I go to run it, the service is still running as the eks-account-role. I'm assuming the default credential provider chain will pick up the AWS profile defined in the config and use that method for authentication, but it doesn't seem to be happening. However, when I try this with the AWS CLI it works as expected (I have properly setup the other role to be assumed by the eks role).
I have found a workaround where if I specify to use profile credentials explicitly it works as expected.
stsClient = StsClient.builder()
.credentialsProvider(ProfileCredentialsProvider.create())
.build();Now this isn't a big deal as it is a small amount of additional code, but I am curious what the SDK is doing under the hood and why it is different from the CLI. A couple things to consider, EKS seems to be injecting two environment variables into the container that could be effecting the behavior AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. However, I am running the CLI from within the container with the same variables present.
Thoughts?
rod-dvla
commented
3 years ago @imcheck make sure you have
aws-java-sdk-stsdependency packaged for your application as well.Not really.
When I try to run s3 client, it says WebIdentityCredentialProvider needs
stsbeen loaded into class path So I searched and getStsWebIdentityCredentialsProviderFactory.javaThen I included
<dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>sts</artifactId> </dependency>It just works :)
Surprised that this works, but it does. Any idea why the separate declaration of sts is required?
StealthyDev
commented
3 years ago I still get an error trying to connect to Secrets Manager from EKS using Service Account role. I'm using sdk v2.16.52.
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Not authorized to perform sts:AssumeRoleWithWebIdentity (Service: Sts, Status Code: 403, Request ID: 13e02bf6-082c-47a2-ace4-900b02792b72, Extended Request ID: null), ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112) at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104) at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.secretsmanager.DefaultSecretsManagerClient.getSecretValue(DefaultSecretsManagerClient.java:1098) at com.hlpp.csw.util.SecretManagerUtil.getSecretFromSM(SecretManagerUtil.java:51) at com.hlpp.csw.config.ShiroCustomEnvironment.setIniOverrides(ShiroCustomEnvironment.java:23) at com.hlpp.csw.config.ShiroCustomEnvironment.init(ShiroCustomEnvironment.java:15) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:45) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:40) at org.apache.shiro.web.env.EnvironmentLoader.createEnvironment(EnvironmentLoader.java:313) at org.apache.shiro.web.env.EnvironmentLoader.initEnvironment(EnvironmentLoader.java:139) at org.apache.shiro.web.env.EnvironmentLoaderListener.contextInitialized(EnvironmentLoaderListener.java:58) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5172) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:692) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1023) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:824) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:474) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1611) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:829) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:433) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
llnformer
commented
3 years ago still happens
SDK version: software.amazon.awssdk:ssm:2.17.19
JDK version used: Java 11
Operating System and version: openjdk:11.0.11-jdk-slim
EKS version: 1.21only after I added software.amazon.awssdk:sts:2.17.19 fixed
itowk
commented
2 years ago Hi, We're still on sdk v1, anyone who's able to workaround this on v1? Using the following dependencies:
madorb
commented
2 years ago @itowk. i don't know if it'll solve your issue, but definitely don't use misaligned versions of the sdk. there's a bom for a reason.
maufranchini
commented
11 months ago Hi, We're still on sdk v1, anyone who's able to workaround this on v1? Using the following dependencies:
- aws-java-sdk-core (v1.12.239)
- aws-java-sdk-sns (v1.11.125)
Adding the latest v1 version (1.11.1034 today's date) for the aws-java-sdk-sts dependency solved my issue (no code needed to be touched using the method DefaultAWSCredentialsProviderChain(). I hope it helps
EKS IAM Service Account Role introduces a new environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and based on the documentation on these two pages, the Java SDK should use "AWS_WEB_IDENTITY_TOKEN_FILE" for credentials if exists.
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
I have tried the latest Java SDK 2.9.19 and it doesn't seem to work. There is a similar discussion on DotNet SDK at here:
https://github.com/aws/aws-sdk-net/issues/1413
I couldn't find docs saying "AWS_WEB_IDENTITY_TOKEN_FILE" is in the list of credentials chain for SDK Java 2. I am wondering if this is implemented or not.
Expected Behavior
Based on the EKS doc link above, Java SDK should recognize the environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and use it to call AssumeRoleWithWebIdentity for access/secret/session tokens.
Current Behavior
I am getting Access Denied with Java SDK in a correctly setup EKS pod with service account. In the same pod, I was able to run
aws s3 lsand it worked, which means the token is correct.Steps to Reproduce (for bugs)
The code I am using to test is from sample S3 code at here: https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3BucketOps.java
I set up a pod in EKS with service account and IAM role properly setup. In the pod, I used aws cli to test the token and it worked. However, the Java SDK didn't work.
Your Environment