search

aws / aws-sdk-java-v2

The official AWS SDK for Java - Version 2
Apache License 2.0
2.21k stars 854 forks source link

Amazon linux 2023 Failed to load credentials from IMDS(SdkServiceException: Unauthorized) #5706

Closed zhxjdwh closed 47 minutes ago

zhxjdwh commented 2 weeks ago

Describe the bug

I have a java application run in EKS cluster. this application will get some secret from Aws secret manager. When i run this application on Amazon Linux 2(EKS node OS) , everything is ok. But when i run on Amazon Linux 2023(EKS node OS)，I got this error.

[ERROR][2024-11-11 13:36:55.236][][main][org.springframework.boot.SpringApplication] Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cn.com.mycompany.common.awscfg.AwsSecretManagerEnvPostProcessor': Initialization of bean failed; nested exception is java.lang.RuntimeException: cn.com.mycompany.common.awscfg.SecretException: Failed to load credentials from IMDS.
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:628)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:188)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:746)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:731)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292)
    at cn.com.mycompany.PlatformUserApplication.main(PlatformUserApplication.java:46)
Caused by: java.lang.RuntimeException: cn.com.mycompany.common.awscfg.SecretException: Failed to load credentials from IMDS.
    at cn.com.mycompany.common.awscfg.AwsSecretManagerEnvPostProcessor.setEnvironment(AwsSecretManagerEnvPostProcessor.java:62)
    at org.springframework.context.support.ApplicationContextAwareProcessor.invokeAwareInterfaces(ApplicationContextAwareProcessor.java:110)
    at org.springframework.context.support.ApplicationContextAwareProcessor.postProcessBeforeInitialization(ApplicationContextAwareProcessor.java:102)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
    ... 15 common frames omitted
Caused by: cn.com.mycompany.common.awscfg.SecretException: Failed to load credentials from IMDS.
    at cn.com.mycompany.common.awscfg.AwsSecretCfgUtil.getAppSecretValueInternal(AwsSecretCfgUtil.java:90)
    at cn.com.mycompany.common.awscfg.AwsSecretCfgUtil.getAppSecretValueViaConfig(AwsSecretCfgUtil.java:28)
    at cn.com.mycompany.common.awscfg.AwsSecretManagerEnvPostProcessor.setEnvironment(AwsSecretManagerEnvPostProcessor.java:58)
    ... 20 common frames omitted
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to load credentials from IMDS.
    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
    at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:167)
    at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300)
    at software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:448)
    at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:208)
    at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:135)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.resolveCredentials(InstanceProfileCredentialsProvider.java:149)
    at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54)
    at software.amazon.awssdk.services.secretsmanager.auth.scheme.internal.SecretsManagerAuthSchemeInterceptor.lambda$trySelectAuthScheme$4(SecretsManagerAuthSchemeInterceptor.java:132)
    at software.amazon.awssdk.core.internal.util.MetricUtils.reportDuration(MetricUtils.java:77)
    at software.amazon.awssdk.services.secretsmanager.auth.scheme.internal.SecretsManagerAuthSchemeInterceptor.trySelectAuthScheme(SecretsManagerAuthSchemeInterceptor.java:132)
    at software.amazon.awssdk.services.secretsmanager.auth.scheme.internal.SecretsManagerAuthSchemeInterceptor.selectAuthScheme(SecretsManagerAuthSchemeInterceptor.java:81)
    at software.amazon.awssdk.services.secretsmanager.auth.scheme.internal.SecretsManagerAuthSchemeInterceptor.beforeExecution(SecretsManagerAuthSchemeInterceptor.java:61)
    at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$beforeExecution$1(ExecutionInterceptorChain.java:59)
    at java.util.ArrayList.forEach(ArrayList.java:1259)
    at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.beforeExecution(ExecutionInterceptorChain.java:59)
    at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.runInitialInterceptors(AwsExecutionContextBuilder.java:241)
    at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:132)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:67)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:76)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
    at software.amazon.awssdk.services.secretsmanager.DefaultSecretsManagerClient.listSecrets(DefaultSecretsManagerClient.java:1303)
    at cn.com.mycompany.common.awscfg.AwsSdkUtil.isExistSecret(AwsSdkUtil.java:61)
    at cn.com.mycompany.common.awscfg.AwsSdkUtil.isExistsSecretByInstanceMeta(AwsSdkUtil.java:50)
    at cn.com.mycompany.common.awscfg.AwsSecretCfgUtil.isExistsSecret(AwsSecretCfgUtil.java:118)
    at cn.com.mycompany.common.awscfg.AwsSecretCfgUtil.readSecretInMultiRegions(AwsSecretCfgUtil.java:97)
    at cn.com.mycompany.common.awscfg.AwsSecretCfgUtil.getAppSecretValueInternal(AwsSecretCfgUtil.java:66)
    ... 22 common frames omitted
Caused by: software.amazon.awssdk.core.exception.SdkServiceException: Unauthorized
    at software.amazon.awssdk.core.exception.SdkServiceException$BuilderImpl.build(SdkServiceException.java:276)
    at software.amazon.awssdk.regions.util.HttpResourcesUtils.handleErrorResponse(HttpResourcesUtils.java:171)
    at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:132)
    at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.lambda$getSecurityCredentials$3(InstanceProfileCredentialsProvider.java:283)
    at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
    at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getSecurityCredentials(InstanceProfileCredentialsProvider.java:283)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.createEndpointProvider(InstanceProfileCredentialsProvider.java:212)
    at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.refreshCredentials(InstanceProfileCredentialsProvider.java:158)
    ... 50 common frames omitted

Regression Issue

Expected Behavior

same as amazon linux 2

Current Behavior

i got an error SdkServiceException: Unauthorized on amazon linux 2023

Reproduction Steps

java code:

  public static boolean isExistsSecretByInstanceMeta(String regionName, String secretName) {
        Region region = Region.of(regionName);
        SecretsManagerClient client = SecretsManagerClient.builder()
                .credentialsProvider(InstanceProfileCredentialsProvider.builder().build())
                .region(region)
                .build();
        return isExistSecret(client, region, secretName);
    }
    private static boolean isExistSecret(SecretsManagerClient client, Region region, String secretName) {
        ListSecretsRequest getSecretValueRequest = ListSecretsRequest.builder()
                .filters(Filter.builder().key(FilterNameStringType.NAME).values(secretName).build())
                .build();

        ListSecretsResponse getSecretValueResponse;

        try {
            getSecretValueResponse = client.listSecrets(getSecretValueRequest);
        } catch (Exception e) {
            // For a list of exceptions thrown, see
            // https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
            throw e;
        }
        if (getSecretValueResponse.secretList() == null || getSecretValueResponse.secretList().isEmpty()) {
            return false;
        }
        return true;
    }

pom.xml

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>secretsmanager</artifactId>
            <version>2.25.48</version>
        </dependency>

eks node OS:

NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.6.20241010"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"

container OS:

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.25.48

JDK version used

openjdk version "11.0.16" 2022-07-19 OpenJDK Runtime Environment 18.9 (build 11.0.16+8) OpenJDK 64-Bit Server VM 18.9 (build 11.0.16+8, mixed mode, sharing)

Operating System and version

container-OS: PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

debora-ito commented 2 weeks ago

When i run this application on Amazon Linux 2(EKS node OS) , everything is ok. But when i run on Amazon Linux 2023(EKS node OS)，I got this error.

If the same code and same SDK version works in one environment but not the other, this indicates an environment misconfiguration, and probably not an issue with the SDK.

According to the EC2 User Guide, 401 - Unauthorized means the GET request used an invalid token, and it's recommended to generate a new one.

Can you run the IMDSv2 curl commands from the instance? https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html#instance-metadata-retrieval-examples

github-actions[bot] commented 4 days ago

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.