search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.66k stars 3.92k forks source link

(assets): Publish command always looks for credentials in `.aws/credentials` #12696

Open ferdingler opened 3 years ago

ferdingler commented 3 years ago

Running the following command cdk-assets publish --path cdk.out/Frontend.assets.json fails on my computer because I don't have credentials hardcoded in my .aws/credentials file. 

Error: ENOENT: no such file or directory, open '/Users/<myuser>/.aws/credentials'
    at Object.openSync (fs.js:476:3)
    at Object.readFileSync (fs.js:377:35)
    at Object.readFileSync (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/util.js:95:26)
    at IniLoader.parseFile (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/shared-ini/ini-loader.js:6:47)
    at IniLoader.loadFrom (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/shared-ini/ini-loader.js:56:30)
    at Config.region (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/node_loader.js:101:36)
    at Config.set (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/config.js:514:39)
    at Config.<anonymous> (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/config.js:349:12)
    at Config.each (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/util.js:512:32)
    at new Config (/usr/local/lib/node_modules/cdk-assets/node_modules/aws-sdk/lib/config.js:348:19) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/Users/<myuser>/.aws/credentials'
}

Reproduction Steps

Run cdk-assets publish without having credentials in .aws/credentials.

What did you expect to happen?

I expect cdk-assets to let the aws-sdk use the DefaultCredentialsProvider so that my credentials can be picked up from the environment variables. I don't want to have harcoded credentials in my .aws/credentials file.

What actually happened?

cdk-assets fails to find credentials.

Environment

This is :bug: Bug Report

ferdingler commented 3 years ago

Peeking at the source code, it seems that this was done on purpose as the comments suggest: https://github.com/aws/aws-cdk/blob/eacd2f7ea67c83d50c839acf29fbe953ae49d987/packages/cdk-assets/bin/publish.ts#L69

NGL321 commented 3 years ago

Hey @ferdingler,

Looks like you are correct that this was an intentional decision. That said, your point about using cached credentials also seems like reasonable use-case. So I am marking this as a feature-request rather than bug so it gets added to the list of potential changes to assets.

Alternatively (if @eladb thinks this is a legitimate feature to add), you could submit a PR with the update.

😸 😷

ferdingler commented 3 years ago

Happy to submit a PR if necessary. This package (cdk-assets) would be very useful for my team.

eladb commented 3 years ago

I don't see a problem with enabling this in cdk-asset, assigning to @rix0rrr for more feedback. I think we will happily accept a contribution.

aramfe commented 3 years ago

Any update on that? This issue is right now forcing me to implement a very ugly workaround in my CI defined jobs using cdk-assets.

awilkins commented 2 years ago

Currently the error seems to have changed (this is in a CodeBuild job), the manifest has two assets.

I have set AWS_SDK_LOAD_CONFIG=1 ; as per above, the emission of these messages in the output isn't helpful because the DefaultAwsClient class forces this value to 1 regardless.


> cdk-assets "publish" "-p" "./cdk.out/RegionalWafAclsStack.assets.json"

info   : [0%] start: Publishing 6b633a26e515f5c5998c8c57add78b783a957696f1b15f82230e5ddcfbc4653f:current_account-eu-west-2
error  : [50%] fail: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
info   : [50%] start: Publishing b0e2ca06e8d2976356673d8e3ba28a5f21339b212f6e6e0ecbc5e804fa547b39:current_account-eu-west-2
error  : [100%] fail: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
Failure: CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
    at Request.extractError (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
Failure: CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
    at Request.extractError (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/codebuild/output/src037991949/src/node_modules/cdk-assets/node_modules/aws-sdk/lib/sequential_executor.js:116:18)```
hansbarnard commented 2 years ago

This is still not fixed. I get the same error when using CDK pipeline component when deploying to multiple regions. The FileAsset publishing stage that runs with CodeBuild fails. I cannot see any workaround. Can't find any other resolutions on the web either. Is anyone even using this software? AWS's response here is pretty disappointing and half-hearted.

dimitarz commented 2 years ago

This was super annoying and I spent a day debugging it. cdk-assets should work out of the box with AWS credentials like every other AWS tools. The way I worked around this in CodeBuild is I added the following to my build spec:

      post_build: {
            commands: [
              'export CREDS=$(curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)',
              'export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r .AccessKeyId)',
              'export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r .SecretAccessKey)',
              'export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r .Token)',
              'for asset in $(ls cdk.out/*assets.json); do npx cdk-assets publish -p $asset || exit 1; done'
            ]
          }
zMeadz commented 2 years ago

For others that run into Pipeline (or remote) cdk-asset publishing / auth issues:

As referenced in packages/aws-cdk/lib/api/aws-auth/sdk.ts, makeDetailedException, the error Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 is generic.

I ran into the same error, but my pipeline did not have permissions to assume the roles responsible for publishing. The fix was the following:

import { PipelineProject } from '@aws-cdk/aws-codebuild';
import { EnvironmentPlaceholders } from '@aws-cdk/cx-api';
import { DefaultStackSynthesizer } from '@aws-cdk/core';
import { Effect, PolicyStatement } from '@aws-cdk/aws-iam';

const assetsPipelineProject: PipelineProject;
...

let fileAssetRole = DefaultStackSynthesizer.DEFAULT_FILE_ASSET_PUBLISHING_ROLE_ARN
    .replace('${Qualifier}', DefaultStackSynthesizer.DEFAULT_QUALIFIER);

fileAssetRole = EnvironmentPlaceholders.replace(fileAssetRole, {
    accountId: <awsAccount>,
    region: <awsRegion>,
    partition: Aws.PARTITION,
});

assetsPipelineProject.addToRolePolicy(
  new PolicyStatement({
    effect: Effect.ALLOW,
    actions: ['sts:AssumeRole'],
    resources: [fileAssetRole],
  })

If you are publishing docker images, then create a role using DefaultStackSynthesizer.DEFAULT_IMAGE_ASSET_PUBLISHING_ROLE_ARN and add it to the policy's resources.

aaa-khoa commented 2 years ago

This was super annoying and I spent a day debugging it. cdk-assets should work out of the box with AWS credentials like every other AWS tools. The way I worked around this in CodeBuild is I added the following to my build spec:

      post_build: {
            commands: [
              'export CREDS=$(curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)',
              'export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r .AccessKeyId)',
              'export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r .SecretAccessKey)',
              'export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r .Token)',
              'for asset in $(ls cdk.out/*assets.json); do npx cdk-assets publish -p $asset || exit 1; done'
            ]
          }

still getting the same error after trying this, did you do anything else in your build? but i agree that it should work the same as the other CDK commands.

HFR1994 commented 1 year ago

I know it's an old thread, but I encountered the same error. Any fix?

[Container] 2023/10/16 19:01:58 Running command tmpfile=/tmp/aws-session-file
[Container] 2023/10/16 19:01:58 Running command aws sts assume-role --role-arn $CDK_ROLE --role-session-name `whoami`-`date +%d%m%y`-session > $tmpfile
[Container] 2023/10/16 19:02:13 Running command export AWS_ACCESS_KEY_ID=`cat $tmpfile|jq -c '.Credentials.AccessKeyId'|tr -d '"'`
[Container] 2023/10/16 19:02:14 Running command export AWS_SECRET_ACCESS_KEY=`cat $tmpfile |jq -c '.Credentials.SecretAccessKey'|tr -d '"'`
[Container] 2023/10/16 19:02:14 Running command export AWS_SESSION_TOKEN=`cat $tmpfile|jq -c '.Credentials.SessionToken'|tr -d '"'`
[Container] 2023/10/16 19:02:14 Running command aws sts get-caller-identity
{
    "UserId": "AROAUPJJK77UOV3FKMFH6:root-161023-session",
    "Account": "307713736680",
    "Arn": "arn:aws:sts::307713736680:assumed-role/cdk-hnb659fds-file-publishing-role-307713736680-us-east-1/root-161023-session"
}
[Container] 2023/10/16 19:02:14 Running command cdk-assets publish -p assets.json
info   : [0%] start: Publishing b7f33614a69548d6bafe224d751a7ef238cde19097415e553fe8b63a4c8fd8a6:307713736680-us-east-1
error  : [50%] fail: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
....
Failure: CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
....

Any luck?

jonodrew commented 1 year ago

For others that run into Pipeline (or remote) cdk-asset publishing / auth issues:

As referenced in packages/aws-cdk/lib/api/aws-auth/sdk.ts, makeDetailedException, the error Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 is generic.

I ran into the same error, but my pipeline did not have permissions to assume the roles responsible for publishing. The fix was the following:

import { PipelineProject } from '@aws-cdk/aws-codebuild';
import { EnvironmentPlaceholders } from '@aws-cdk/cx-api';
import { DefaultStackSynthesizer } from '@aws-cdk/core';
import { Effect, PolicyStatement } from '@aws-cdk/aws-iam';

const assetsPipelineProject: PipelineProject;
...

let fileAssetRole = DefaultStackSynthesizer.DEFAULT_FILE_ASSET_PUBLISHING_ROLE_ARN
    .replace('${Qualifier}', DefaultStackSynthesizer.DEFAULT_QUALIFIER);

fileAssetRole = EnvironmentPlaceholders.replace(fileAssetRole, {
    accountId: <awsAccount>,
    region: <awsRegion>,
    partition: Aws.PARTITION,
});

assetsPipelineProject.addToRolePolicy(
  new PolicyStatement({
    effect: Effect.ALLOW,
    actions: ['sts:AssumeRole'],
    resources: [fileAssetRole],
  })

If you are publishing docker images, then create a role using DefaultStackSynthesizer.DEFAULT_IMAGE_ASSET_PUBLISHING_ROLE_ARN and add it to the policy's resources.

I've also run into this issue with a multi-account pipeline. @zMeadz , in the above, is the <awsAccount> the account where the pipeline is, or the account you're trying to deploy into?

zMeadz commented 1 year ago

@jonodrew references the target deployment account

f0ff-s8 commented 1 year ago

How is this still not fixed?

SelenaBostick commented 1 year ago

@zMeadz

let fileAssetRole = DefaultStackSynthesizer.DEFAULT_FILE_ASSET_PUBLISHING_ROLE_ARN .replace('${Qualifier}', DefaultStackSynthesizer.DEFAULT_QUALIFIER);

Is this part replacing the qualifier with something else, or using the same value as the default? Would you mind unpacking what this change does?