"unable to validate AWS credentials" when setting AWS profile via Pulumi config

[jasminen]

What happened?

Up until now we have used env vars to set and use the default AWS provider:

export AWS_REGION=xx-xxx-x
export AWS_PROFILE=yyyyy

Now i am trying to use the Pulumi config instead of env vars, so i've added the following settings to the stack's pulumi config:

config:
  aws:profile: yyyyy
  aws:region: xx-xxx-x

Now i am getting the following errors:

error: unable to validate AWS credentials.
    Details: no valid credential sources for  found.

    Please see
    for more information about providing credentials.

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed

    Make sure you have:

         • Set your AWS region, e.g. `pulumi config set aws:region us-west-2`
         • Configured your AWS credentials as per https://pulumi.io/install/aws.html
         You can also set these via cli using `aws configure`.

I tried to add skipCredentialsValidation: true and skipMetadataApiCheck: false but getting the same result.

Expected Behavior

Should work with pulumi config the same as it worked with the env vars

Steps to reproduce

Use a simple AWS project without setting an explicit AWS provider (using the default one) and setting region and profile with env vars. Then remove env var and set region and profile using pulumi config.

Output of pulumi about

➜ ~ pulumi about CLI Version 3.49.0 Go Version go1.19.3 Go Compiler gc

Plugins NAME VERSION aws 5.24.0 aws 5.24.0 cloudflare 3.6.0 cloudflare 3.3.0 datadog 3.3.0 datadog 3.3.0 docker 3.6.1 docker 3.0.0 kafka 3.4.0 kafka 3.0.1 kubernetes 3.6.0 kubernetes 3.6.0 mysql 3.1.0 mysql 3.0.0 nodejs unknown okta 3.17.0 okta 3.0.0 pagerduty 2.2.0 pagerduty 2.1.1 postgresql 3.6.0 postgresql 3.6.0 rabbitmq 3.2.0 rabbitmq 3.0.0 random 4.10.0 random 4.2.0 spotinst 3.8.0 spotinst 3.8.0 tls 4.8.0 tls 4.0.0

Host OS amazon Version 2 Arch x86_64

This project is written in nodejs: executable='~/.nvm/versions/node/v14.21.0/bin/node' version='v14.21.0'

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[thomas11]

Hi @jasminen, following up on our conversation in Slack here. I tried to reproduce this with a new program and a new stack (via pulumi new aws-go), but wasn't able to. My terminal session is below.

I did go down a rabbit hole for a bit because I tried to repro it with a program using an explicit provider, and there seems to be something wrong there. However, you say you're using the default provider.

As a next step, are you able to add some debug output to this issue? Via pulumi preview --logtostderr --logflow -v=9 2> out.txt.

$ pulumi preview
Diagnostics:
  aws:s3:Bucket (my-bucket):
    error: unable to validate AWS credentials.
    Details: no valid credential sources for  found.
...

$ export AWS_PROFILE=AdministratorAccess-616138583583
$ export AWS_REGION=us-west-2

$ pulumi preview

     Type                 Name                   Plan
 +   pulumi:pulumi:Stack  repro-aws-creds-2-dev  create
 +   └─ aws:s3:Bucket     my-bucket              create

Outputs:
    bucketName: output<string>

Resources:
    + 2 to create

$ pulumi config set aws:profile AdministratorAccess-616138583583

$ pulumi config set aws:region us-west-2

$ unset AWS_PROFILE

$ unset AWS_REGION

$ export -p | rg AWS

$ pulumi preview

     Type                 Name                   Plan
 +   pulumi:pulumi:Stack  repro-aws-creds-2-dev  create
 +   └─ aws:s3:Bucket     my-bucket              create

Outputs:
    bucketName: output<string>

Resources:
    + 2 to create

[thomas11]

Hey @jasminen! Did you get a chance to take another look at this issue?

[squaremo]

Side issue (but possibly relevant): what's up with the missing value in

Details: no valid credential sources for  found.

[SharpEdgeMarshall]

@thomas11 same issue here, we are using this set of ENVs in CI to assume roles through profiles:

AWS_PROFILE: <PROFILE_NAME>
AWS_SDK_LOAD_CONFIG: "true"
AWS_CONFIG_FILE=<PATH_TO_CUSTOM_AWS_CONFIG>

Configs:

skipCredentialsValidation: "true"
skipMetadataApiCheck: "false"

Looks like it breaks when we try to use an explicit AWS Provider passing another profile name

Error: Error: failed to refresh cached credentials, operation error STS: AssumeRole, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, expect HTTP transport, got <nil>

[thomas11]

Side issue (but possibly relevant): what's up with the missing value in

Details: no valid credential sources for  found.

Fixed in #2367, but not related to this issue, unfortunately.

[thomas11]

Hi @SharpEdgeMarshall , I tried to reproduce a few basic scenarios, like setting AWS_PROFILE and in the code creating a provider with a different profile:

        provider, err := aws.NewProvider(ctx, "provider", &aws.ProviderArgs{
            Profile: pulumi.StringPtr("accesskey"),
            Region:  pulumi.StringPtr("us-west-2"),
        })

Everything worked for me. Could you give me more information about your scenario, ideally some sample code to reproduce it?

[thomas11]

Hi @jasminen, since I cannot reproduce the issue, I'll close it soon. Did you get a chance to look at this again?

[SharpEdgeMarshall]

@thomas11 Are you testing it on an AWS instance with an instance profile?

[thomas11]

@SharpEdgeMarshall no, should I?

Can you share a code snippet of how exactly you're creating the explicit provider?

[throttlehead-dev]

@thomas11 I have a theory that this is something to do with how default providers are resolved. Digging around the code for aws.Provider, it seems that if nothing is supplied for say region, it falls back to env variables. It probably first needs to fallback to the configured aws:region, and similarly for profile.

For me this issue seems to arise with VpcPeeringConnection stuff. I wonder if that particular component is doing "quasi" default providers and ignoring aws:profile?

When I create a defaultProvider explicitly using config values, the code seems to run as intended:

const defaultProvider = new aws.Provider(`${stack}-default-provider`, {
    region: getConfig("aws:region") as Region,
    profile: getConfig("aws:profile")
});

    const peerVpcPeeringConnection = new aws.ec2.VpcPeeringConnection(peerConnectionName, {
        vpcId: vpcId,
        peerVpcId: connection.id,
        peerOwnerId: accountId,
        peerRegion: connection.region,
        autoAccept: false,
        tags: {
            Side: "Requester",
            Name: `${peerConnectionName}`
        },
    }, {
        provider: defaultProvider
    });

[thomas11]

Hi @throttlehead-dev, this is actually as intended, although I do agree that it could be documented more clearly. See here:

Configuration file settings are only used by the default provider. If you instantiate a provider object, it will not read values from the stack configuration.

[thomas11]

Since I wasn't able to reproduce the issue and some of the commenters didn't follow up so far, I'll close this issue. Feel free to reopen it if you still think Pulumi has a problem here.

[ttreptow]

If you are getting the operation error ec2imds: GetMetadata, expect HTTP transport, got <nil> error and you are running your code in a container or kubernetes, you might need to adjust the hop limit on metadata service PUT requests. Going through an internal bridge counts as a hop, so your requests are probably getting blocked by the metadata service (hence the nil response).

[timbrammer910]

Hey there, I'd like to reopen this.

I'm experiencing this with pulumi-3.111.0 and pulumi-aws-5.43.0 using the default provider.

We're generating ~/.aws/credentials in our CI pipeline, then running a pulumi refresh.

When setting aws:profile: some-account it fails with

error: Preview failed: unable to validate AWS credentials.
    Details: no valid credential sources for  found.

    Please see
    for more information about providing credentials.

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: i/o timeout

but when setting the env var AWS_PROFILE it works fine.