CLI: No credentials have been configured error for deployment in cn-north-1 region from EC2 instance not supporting IMDSv2

sivakova commented 2 days ago

Describe the bug

I am trying to deploy resources form EC2 instance in cn-north-1 region Metadata token fetch for IMDSv2 fails and it seems like fallback to IMDSv1 doesn't happen. Default region us-east-1 is set in the end and deployment fails.

The issue with credentials in China region started since 2.163.0 aws-cdk version.

build 26-Nov-2024 11:11:13 [11:11:13] [trace] SdkProvider#withAwsCliCompatibleDefaults() build 26-Nov-2024 11:11:13 [11:11:13] Looking up AWS region in the EC2 Instance Metadata Service (IMDS). build 26-Nov-2024 11:11:14 [11:11:14] Unable to retrieve AWS region from IMDS: Error: Error fetching metadata token: TimeoutError: Connection timed out after 1000 ms build 26-Nov-2024 11:11:14 [11:11:14] Unable to determine AWS region from environment or AWS configuration (profile: "default"), defaulting to 'us-east-1' build 26-Nov-2024 11:11:14 [11:11:14] Toolkit stack: CDKToolkit build 26-Nov-2024 11:11:14 [11:11:14] Setting "CDK_DEFAULT_REGION" environment variable to us-east-1

Error message:

build 26-Nov-2024 11:11:17 [11:11:17] red: debug: Need to perform AWS calls for account **, but no credentials have been configured build 26-Nov-2024 11:11:17 [11:11:17] red: debug: Need to perform AWS calls for account **, but no credentials have been configured

Regression Issue

[x] Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The region is set to cn-north-1, credentials is successfully configured, deployment is successful

Current Behavior

The region is set to us-east-1 and credentials is not configured correctly

Reproduction Steps

Deploy any resource from EC2 instance in cn-north-1 region without configured environment variables specifying region, and without configuration file with default profile

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.171.1

Framework Version

No response

Node.js Version

23.3.0

OS

MacOS

Language

TypeScript

Language Version

5.6.3

Other information

No response

ashishdhingra commented 2 days ago

@sivakova Good morning. Below is some high level analysis:

You are probably getting error Unable to retrieve AWS region from IMDS: from code here
This uses MetadataService from @aws-sdk/credential-providers from AWS JavaScript SDK v3.
MetadataService class has optional property ec2MetadataV1Disabled.
AWS CDK's regionFromMetadataService() doesn't set the option ec2MetadataV1Disabled. Hence, fallback to IMDSv1 should occur automatically by AWS SDK for JavaScript v3 (CDK has no control over it).
- It uses endpoint /latest/dynamic/instance-identity/document to load identity document that provides region (along with other data, refer Instance identity documents for Amazon EC2 instances).
- If you refer Retrieve the instance identity document for an Amazon EC2 instance, it has the same URL http://169.254.169.254/latest/dynamic/instance-identity/document for retrieving identity document for both IMDSv1 and IMDSv2.

Could you please check on your side if IMDSv1 is not disabled on your EC2 instance? Refer Get the full benefits of IMDSv2 and disable IMDSv1 across your AWS infrastructure, you may select either of values V1 and V2 (token optional) and V2 only (token required) for Metadata version for an EC2 instance.

Also could you try using the latest version of AWS CDK lib (in addition to AWS CDK CLI)?

Thanks, Ashish

mrgrain commented 2 days ago

@sivakova Are you able to try this with the latest version v2.171.1. We had some major changes to auth since that got releases after 2.163.0.

sivakova commented 2 days ago

@mrgrain yes I am trying with the latest version and the issue is still there

sivakova commented 1 day ago

@ashishdhingra Hello! I have an idea what the problem might be. It's this line here in regionFromMetadataService() fetchMetadataToken() throws error in case of unsupported IMDSv2 and the next function call metadataService.request() which also includes token fetch inside and fallback to IMDSv1 is out of reach

I am not able to check right now whether IMDSv1 is disabled on EC2 or not, but i was able to make request to http://169.254.169.254/ from the the instance and it works as expected

mrgrain commented 1 day ago

@mrgrain yes I am trying with the latest version and the issue is still there

Thanks for the confirmation. Does the error change at all or is it all the same?

sivakova commented 2 hours ago

@mrgrain good day! First it was another error about cross account deployment but it was fixed. (error message: Need to perform AWS calls for account , but the current credentials are for )

Then for some further versions i stopped getting creds error. The exact error (Need to perform AWS calls for account **, but no credentials have been configured) has been noticed since 2.167.1 version

aws / aws-cdk