Open marcusirgens opened 9 months ago
Just want to say thank you, after a long struggle with why --profile XXXX
did not work I found your issue just in time. Removing the sso-session
block and sso_session
params from my configs did resolve it and now I can deploy my multi account pipeline.
Further notes on the topic which may be helpful, I do not have any [default]
block in my .aws/config file and no blocks at all in my .aws/credentials file.
Additionally I found out that if I use the flag --profile
with npm run cdk synth --profile xxx
it does not work at all (it may get eaten by npm ?). If you need to use npm run
you need to set the above mentioned AWS_PROFILE
otherwise if you use a global installed cdk you can use the --profile
flag.
There's a handful of possible issues here.
@indrora were you able to reproduce this?
Is the account that the CDK is trying to use (that is, the AWS Account that you've SSO logged in as) bootstrapped?
@indrora Do you mean with CDK?
$ AWS_PROFILE=my-org-dev cdk bootstrap
# Output:
# ⏳ Bootstrapping environment aws://123456789012/eu-north-1...
# ❌ Environment aws://123456789012/eu-north-1 failed bootstrapping: Error: Need to perform AWS calls for account 123456789012, but no credentials have been configured
# at SdkProvider.forEnvironment (/Users/buggs/node/n/lib/node_modules/aws-cdk/lib/index.js:367:13075)
# at async _BootstrapStack.lookup (/Users/buggs/node/n/lib/node_modules/aws-cdk/lib/index.js:454:20535)
# at async Bootstrapper.modernBootstrap (/Users/buggs/node/n/lib/node_modules/aws-cdk/lib/index.js:455:1084)
# at async /Users/buggs/node/n/lib/node_modules/aws-cdk/lib/index.js:459:2104
# at async Promise.all (index 0)
# at async CdkToolkit.bootstrap (/Users/buggs/node/n/lib/node_modules/aws-cdk/lib/index.js:459:1949)
# at async exec4 (/Users/buggs/node/n/lib/node_modules/aws-cdk/lib/index.js:512:53102)
#
# Need to perform AWS calls for account 123456789012, but no credentials have been configured
Without sso_session
in $AWS_CONFIG_FILE
(see https://github.com/aws/aws-cdk/issues/27265#issuecomment-1734559718):
$ AWS_PROFILE=my-org-dev cdk bootstrap
# Output:
# ⏳ Bootstrapping environment aws://123456789012/eu-north-1...
#Trusted accounts for deployment: (none)
#Trusted accounts for lookup: (none)
#Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
#
# ✨ hotswap deployment skipped - no changes were detected (use --force to override)
#
# ✅ Environment aws://123456789012/eu-north-1 bootstrapped (no changes).
Is the account that you have used for SSO able to AssumeRole against the role that the CDK has created to run deployments?
Yes, as far as I can tell (the bottom part is a script that assumes all roles with CDK in their names):
$ AWS_PROFILE=my-org-dev aws sts get-caller-identity --output json
# Output:
# {
# "UserId": "ARARANDOMREDACTED:buggs@iterate.no",
# "Account": "123456789012",
# "Arn": "arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_AdministratorAccess_randomsuffix/buggs@iterate.no"
# }
# This script lists all roles with "cdk" in the name, and runs aws sts assume-role for those, and prints the assumed role.
$ AWS_PROFILE=my-org-dev aws iam list-roles --output json | \
jq '.Roles[] | .Arn | select(. | ascii_downcase | contains("cdk")) | .' --raw-output0 | \
AWS_PROFILE=my-org-dev xargs -0n1 -I'{}' aws sts assume-role --role-arn '{}' --role-session-name 'demo-github' --output json | \
jq '.AssumedRoleUser.Arn'
# Output:
# # An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_AdministratorAccess_9291ebc2c934fe48/buggs@iterate.no is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789012:role/cdk-hnb659fds-cfn-exec-role-123456789012-eu-central-1
#
# An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_AdministratorAccess_9291ebc2c934fe48/buggs@iterate.no is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789012:role/cdk-hnb659fds-cfn-exec-role-123456789012-eu-north-1
#
# An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::123456789012:assumed-role/AWSReservedSSO_AdministratorAccess_9291ebc2c934fe48/buggs@iterate.no is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789012:role/cdk-hnb659fds-cfn-exec-role-123456789012-us-east-1
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-deploy-role-123456789012-eu-central-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-deploy-role-123456789012-eu-north-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-deploy-role-123456789012-us-east-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-file-publishing-role-123456789012-eu-central-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-file-publishing-role-123456789012-eu-north-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-file-publishing-role-123456789012-us-east-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-image-publishing-role-123456789012-eu-central-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-image-publishing-role-123456789012-eu-north-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-image-publishing-role-123456789012-us-east-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-lookup-role-123456789012-eu-central-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-lookup-role-123456789012-eu-north-1/demo-github"
# "arn:aws:sts::123456789012:assumed-role/cdk-hnb659fds-lookup-role-123456789012-us-east-1/demo-github"
The three roles I cannot assume have the following trust policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "cloudformation.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
To be 100 % clear:
With the following config, I can perform AWS_PROFILE=my-org-dev aws sso login
with no issues and I can assume roles, but I cannot use CDK.
[profile my-org-dev]
sso_session = Admin in My Org
sso_account_id = <REDACTED>
sso_role_name = AdministratorAccess
region = eu-north-1
[sso-session 'Admin in My Org']
sso_start_url = https://<REDACTED>.awsapps.com/start#
sso_region = eu-central-1
sso_registration_scopes = sso:account:access
This is my entire AWS_CONFIG_FILE
, there are no other sections in it.
I've also removed files from ~/.aws/cli/cache
and ~/.aws/sso/cache
to double check that I'm not using any "hidden" credentials, which successfully gave me the following output:
Error loading SSO Token: Token for Admin in My Org does not exist
When removing the sso_session
parameter, and moving sso_start_url
and sso_region
up to profile, making this my entire AWS_CONFIG_FILE
, I can use CDK normally:
[profile my-org-dev]
# sso_session = Admin in My Org
sso_account_id = <REDACTED>
sso_role_name = AdministratorAccess
region = eu-north-1
sso_start_url = https://<REDACTED>.awsapps.com/start#
sso_region = eu-central-1
# [sso-session 'Admin in My Org']
# sso_registration_scopes = sso:account:access
The output of aws sts get-caller-identity --output text --query Arn
is byte-for-byte the same when using either of these configurations. The problem is that the first one is the one generated by aws configure sso
.
@peterwoodworth I don't have an account set up with SSO right now, so I wasn't able to do a proper repro.
Some notes, however:
Discussion #21316 might be of interest: If you're behind an SSL crusher like ZScaler so that Corporate IT can do SSL inspection, the AWS client the CDK uses just wanders into the sun because it doesn't support the custom CA bundle. I don't know if that's the case here, but it's a plausible lead.
I have found numerous issues with AWS SSO and CDK discussed by other customers(1) and in our own ticket tracker (#23520), and found one customer who found a... While not great workaround, a workaround(2). There is a known suggestion on re:post to just do it the hard way(3) for some applications.
PR #19454 was supposed to fix this need back in 2022, but I don't know how well that's worked in the long run. There was also work in #23702 to handle an edge case in SSO profiles.
I had this issue and it turned out I had some old entries in my ~/.aws/config
that looked like
[name]
region = ap-southeast-2
Removing all of those (and leaving the newer [profile name]
and [sso-session name]
entries did the trick
same issues for me, please fix. Workaround indeed to change profile name to other than default
same issues for me, please fix. Workaround indeed to change profile name to other than default
Clarifying that this is not the issue I've raised here, nor the fix for it - this issue seems to be that CDK does not understand the SSO session field.
The "workaround" seems to be to not enter anything when prompted for
SSO session name (Recommended):
while running aws configure sso
, which sets up SSO in "legacy" mode, which works fine.
Not one of the solutions proposed work for me when trying to quickly deploy CDK stacks into multiple accounts. Whichever one I configure and use, will cause the other to error out with the "need to perform for A, but current creds are for B".
The unacceptable work-around involves a rigmarole of cleaning/purging of credential and auth caches. Then switch to the other, deploy, and of course get the same error when trying to use the other account, except this time with "need to perform for B, but current creds are for A".
Somewhat ironically, this used to work without issues when using different SSO providers (from Okta to Jumpcloud).
There's a chance this old issue is related: https://github.com/aws/aws-cdk/issues/24744
To add another data point: I'm also using aws sso login
, and I will randomly get that error with CDK commands every now and then, but I can immediately run the same command again (cdk deploy)
and it will work, so my SSO session was still valid.
Sounds like a bug in the CLI that sometimes randomly fails to return SSO credentials to CDK.
Sounds like a bug in the CLI
The CLI does not provide credentials to the CDK, not directly. Running aws sso login
gets an Identity Center token and caches it on the file system. The AWS SDKs load that token and exchange it for IAM credentials. If the CLI works after using aws sso login
, e.g. aws sts get-caller-identity
, the CLI has correctly cached the token. Any further issues would be in the CDK or its dependencies.
The issue is probably somewhere in the JavaScript SDK. As I understand it, different parts of the CDK use v2 and v3 of the JS SDK, and Identity Center support in v2 was very late in coming, so I wouldn't be surprised if it also has issues with the change to session-based profiles and refresh tokens. Potentially this: https://github.com/aws/aws-sdk-js/issues/4441
What can I do to help diagnose the source of the bug? Is there a debug log I can turn on? I could post the logs when the problem reproduces itself.
After much messing about, I found out that I could get it to work by removing any spaces in my sso-session name and removing the single quotes around the sso-session name (not certain if this was necessary or not)
So
[profile my-org-dev]
sso_session = Admin in My Org
...
[sso-session 'Admin in My Org']
...
Could be changed to:
[profile my-org-dev]
sso_session = Admin-in-My-Org
...
[sso-session Admin-in-My-Org]
...
Seems like other config readers have run into this too (ruby, go). If I'm following the code right here the correct fix would need to be in https://github.com/smithy-lang/smithy-typescript. Though it seems like the aws cli is actually generating configuration that violates the spec. It should probably prevent using sso session names with spaces in them.
You can use letters, numbers, hyphens ( - ), and underscores ( _ ), but no spaces.
I'm having this issue even without spaces or quotes in my session name.
[profile test]
sso_session = myorg
sso_account_name = Test
sso_account_id = <redacted>
sso_role_name = Admin
region = eu-west-1
output = json
[sso-session myorg]
sso_region = us-east-1
sso_start_url = https://<redacted>.awsapps.com/start
same issues for me, please fix. Workaround indeed to change profile name to other than default
Clarifying that this is not the issue I've raised here, nor the fix for it - this issue seems to be that CDK does not understand the SSO session field.
The "workaround" seems to be to not enter anything when prompted for
SSO session name (Recommended):
while running
aws configure sso
, which sets up SSO in "legacy" mode, which works fine.
This works to me
Describe the bug
When setting up a project from scratch with SSO credentials, CDK fails to authorize.
The steps I'm following are as follows:
aws configure sso
aws sso login --profile my-org-dev
bin/{stack}.ts
AWS_PROFILE=my-org-dev cdk deploy
I receive the following error message:
Expected Behavior
I expect CDK to be compatible with credentials generated with
aws configure sso
andaws sso login
.Current Behavior
When deploying, I get the following output:
Getting my current session details:
This is what my
AWS_CONFIG_FILE
looks like:Reproduction Steps
I install
aws-cdk
and check the version:AWS_CONFIG_FILE
and delete the existing file:I go to the SSO start URL for my organization, https://acme.awsapps.com/start. Under the account I want to use, I click Command line or programmatic access, where I read the following instructions:
In my shell, I configure the AWS CLI using
aws configure sso
:In my shell, I verify that I have a valid session:
I initialize a CDK project in a new directory:
I attempt to deploy the stack, which fails because CDK cannot determine which account to use:
I follow the instructions in
bin/{the application name}.ts
, as indicated in the output from the failingcdk deploy
, and add:I attempt to deploy the stack, which fails because CDK cannot find the credentials:
If I remove the
sso-session
block and move some settings up to themy-org-dev
profile, I am able to deploy:$AWS_CONFIG_FILE
:I attempt to deploy the stack, which now succeeds:
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.97.0 (build d7cf3be)
Framework Version
No response
Node.js Version
18.14.0
OS
macOS 13.4 (22F66)
Language
Typescript
Language Version
~5.2.2
Other information
Similar issues
This seems to be somewhat related to a couple of other issues.
25870 has the same "credential process" issue, but is not related to SSO
23520 has the exact same issue. Following the instructions from https://github.com/aws/aws-cdk/issues/23520#issuecomment-1369308510 does not give the same results (
aws --profile my-org-dev sts get-caller-identity
succeeds, butnpx cdk diff --profile my-org-dev
fails like above). As far as I can tell, my$AWS_CONFIG_FILE
has the same properties.20935 has recent comments from people experiencing similar issues
24744 has similar issues with resolving the account ID for SSO credentials