Open kirnberger1980 opened 1 year ago
Thanks for reporting.
Can you run this in the pod from Account A?
aws --profile AccountB sts get-caller-identity
Can you share a minimal CDK code that provisions this environment so I can try reproduce it in my account?
Thanks for the quick reply.
Here is the result of the aws cli response:
{
"UserId": "AXXXXXXXXXX:botocore-session-1686146634",
"Account": "<AccountB>",
"Arn": "arn:aws:sts::<AccountB>:assumed-role/RoleB/botocore-session-1686146634"
}
This is, what we expected, as setting env variable AWS_PROFILE or using --profile should return the identical response.
Here is some sample code tested, where we could reproduce the issue:
bin-File:
#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdkStack } from '../lib/cdk-stack';
const app = new cdk.App();
new CdkStack(app, 'CdkStack', {
env: { account: "<AccountB>",region: "eu-central-1" }
});
lib-File:
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
export class CdkStack extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const vpc = cdk.aws_ec2.Vpc.fromLookup(this, "MyVPC", {
vpcName: "MyVpcName"
});
new cdk.aws_ec2.SecurityGroup(this, "MyNewSecurityGroup", {
vpc,
description: "This will secure my resources"
});
}
}
Kind Regards, Marcell
Any news on this issue?
This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.
I am facing a similar issue. An update would be greatly appreciated.
Whats going on with this issue? We are facing the same issue but sounds like a common workflow!
Is this a recent thing? I just now saw this. CDK worked totally fine then suddenly it's now failing with this error and in the verbose log I see profile xxx did not include credential process
. I have the profile name exported in the AWS_PROFILE
variable. I reset the terminal and ran aws sso login
again, but nothing helped. Removal of cdk.out
folda also didn't help. It's as if it broke as a result of some change in the cloud since it broke today after 16:00 and before that worked just fine.
UPDATE: I managed to fix this by updating the awscliv2 to the latest version, doing aws sso logout
then aws sso login
and then it worked. So might be something to do with token timings, and not CDK per se. Still, strange. AWS CLI commands worked just fine.
Updating: this seems to have something to do with timing. This happens if the session expires. Once it does, CDK will break and no amount of logging out and back in will help. Only waiting seems to do the trick. I have also tried clearing context, but still nothing. After waiting for hours, and then logging out and back in the problem goes away until the next day, but that is hardly a "solution". Is anyone looking into this?
I think the problem is in web_identity_token_file
.
In my case with AWS_PROFILE=oidc
environment variable set and the following content in ~/.aws/config
file
[profile oidc]
role_arn=arn:aws:iam::012345678901:role/gitlab-deploy
web_identity_token_file=/tmp/web_identity_token
command aws sts get-caller-identity
works as expected.
But npx cdk diff -vv
fails with
...
[16:42:45] Determining if we're on an EC2 instance.
[16:42:45] Does not look like an EC2 instance.
[16:42:45] Toolkit stack: CDKToolkit
[16:42:45] Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
[16:42:45] Resolving default credentials
[16:42:45] Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile oidc did not include credential process
[16:42:45] context: {
...
So I ended up assuming role and setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables with command
export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
$(aws sts assume-role-with-web-identity
--role-arn ${ROLE_ARN}
--role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
--web-identity-token ${GITLAB_OIDC_TOKEN}
--duration-seconds 3600
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
--output text))
I'm pretty sure the issue here is that our current credential chain does not support web_identity_token_file
inside an ini file, so the credential chain here fails on profileCredentials
because of the presence of web_identity_token_file
and then again on SsoCredentials
because none are setup, as expected. The last credential provider is ProcessCredentials
, which will fail here, also as expected. Since it is the last in the chain, that's the error that actually gets surfaced, which is super misleading.
We are currently in the process of upgrading from sdkv2 to sdkv3, at which point web_identity_token_file
will be supported and you should no longer experience this error. You can follow the progress in https://github.com/aws/aws-cdk/pull/31702, if you'd like.
Describe the bug
We want to use the CDK CLI with Gitlab Runner, which is running on EKS in Account A. cdk deploy is executed for Account B (Cross-Account Deployment).
CLI Command fails with:
We are using IRSA (IAM Role for Service Accounts), further called Role A, for the Gitlab Runner in Account A. In Account B there is an IAM Role (Role B), which can be assumed by Role A. RoleB can assume all cdk-provisioned IAM Roles in that Account (created by cdk bootstrap).
Profile file (stored in /configmaps/awsconfig):
We have set environment variables AWS_PROFILE and AWS_CONFIG_FILE:
AWS CLI is working fine with above profile file and the environment variables:
Assuming the cdk-lookup Role from RoleB with AWS CLI works fine.
By the way: cdk also seems not to respect AWS_PROFILE environment variable as it is using the default role (it could be, that the session token by IRSA is used), when NOT using --profile.
Expected Behavior
We expect that AWS CDK CLI is acting identically to AWS CLI when using AWS profiles for Cross-Account Deployments. Environment Variable AWS_PROFILE should be respected.
Current Behavior
The CDK CLI fails that the cdk-Lookup-Role cannot be assumed, because "Profile did not include credential process". When you don't specify --profile, then default-Profile (or session token created by OIDC Provider of IAM) is used.
Reproduction Steps
You need to run a pod in EKS:
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.78.0
Framework Version
2.78.0
Node.js Version
v18.16.0
OS
node:18-alpine
Language
Typescript
Language Version
3.9.7
Other information
No response