cdktf is unable to resolve AWS local credentials

howdoicomputer commented 9 months ago

Expected Behavior

cdktf deploy is able to resolve ~/.aws/credentials and ~/.aws/config when ran

Actual Behavior

I encountered an error while going through the tutorial for deploying applications to AWS. This one here: https://developer.hashicorp.com/terraform/tutorials/cdktf/cdktf-build

When running cdktf deploy cdktf is unable to resolve the location to my ~/.aws/credentials and ~/.aws/config files.

For example:

aws-cdktf-example  ╷
                   │ Error: No valid credential sources found
                   │ 
                   │   with provider["registry.terraform.io/hashicorp/aws"],
                   │   on cdk.tf.json line 24, in provider.aws[0]:
                   │   24:

I validated that my credentials are correct as I'm able to use the awscli to query resources.

I also created a simple, plain Terraform setup and it was able to run just fine:

provider "aws" {}

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource aws_instance "foobar" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"
}

I also tried setting AWS_SHARED_CREDENTIALS_FILE and AWS_SHARED_CONFIG_FILE environment variables and that didn't work either. I also tried setting the file locations for the provider in my cdktf code and that also didn't work. I also tried to force cdktf to use the version 4 of the AWS provider but couldn't get it to do that either - it just installed both 4 and 5 alongside each other and defaulted to 5. The above Terraform example used the same AWS provider version: 5.20.1.

Steps to Reproduce

Create an AWS user and generate keys for that user.
Run aws configure and specify those keys.
Go through the Build AWS Infrastructure with CDK for Terraform tutorial.
Watch it fail when it can't find ~/.aws/credentials or ~/.aws/config.

Versions

language: typescript cdktf-cli: 0.18.2 node: v18.18.0 cdktf: 0.18.2 constructs: 10.3.0 jsii: null terraform: 1.6.0-dev arch: x64 os: linux 6.1.56

Providers

[howdoicomputer@framework:~/workspace/aws-cdktf-example]$ cdktf provider list ┌───────────────┬──────────────────┬─────────┬────────────┬─────────────────────┬─────────────────┐ │ Provider Name │ Provider Version │ CDKTF │ Constraint │ Package Name │ Package Version │ ├───────────────┼──────────────────┼─────────┼────────────┼─────────────────────┼─────────────────┤ │ aws │ 4.67.0 │ │ ~> 4.0 │ │ │ ├───────────────┼──────────────────┼─────────┼────────────┼─────────────────────┼─────────────────┤ │ aws │ 5.20.1 │ ^0.18.0 │ │ @cdktf/provider-aws │ 17.0.10 │ └───────────────┴──────────────────┴─────────┴────────────┴─────────────────────┴─────────────────┘

There are two providers because I was trying to use the previous provider but couldn't figure out how to force cdktf to use version 4 of the AWS provider.

Gist

https://gist.github.com/howdoicomputer/601137636cdc079799e1c2c7daee2b64

Possible Solutions

I haven't tried hard coding secrets in cdktf yet but I really don't want to do that.

Workarounds

No.

Anything Else?

No response

References

No response

Help Wanted

[X] I'm interested in contributing a fix myself

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

howdoicomputer commented 9 months ago

I went further down the rabbit hole with this.

I was able to figure out how to lock the AWS provider. I needed to lock the package version for the AWS provider bindings to 14.0.4 - otherwise the latest version of the bindings just does its own thing without bubbling up an error to the user.

So with an older, hopefully more stable version of the AWS provider:

aws-cdktf-example  Initializing provider plugins...
                   - Finding hashicorp/aws versions matching "4.66.1"...
aws-cdktf-example  - Installing hashicorp/aws v4.66.1...
aws-cdktf-example  - Installed hashicorp/aws v4.66.1 (signed by HashiCorp)

And with my provider config statically set:

    new AwsProvider(this, "AWS", {
      region: "us-west-2",
      profile: "default",
      sharedConfigFiles: ["$HOME/.aws/config"],
      sharedCredentialsFiles: ["$HOME/.aws/credentials"],
    });

I'm getting:

                   │ Error: configuring Terraform AWS Provider: failed to get shared config profile, default
                   │ 
                   │   with provider["registry.terraform.io/hashicorp/aws"],
                   │   on cdk.tf.json line 31, in provider.aws[0]:
                   │   31:

Again, this works with standard terraform and the awscli so it has to be cdktf that is doing something weird. Or something is incredibly wacky about my local dev environment.

howdoicomputer commented 9 months ago

Contents of ~/.aws/credentials

[default]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED

Contents of ~/.aws/config:

[howdoicomputer@framework:~/workspace/aws-cdktf-example]$ cat ~/.aws/config
[default]
region = us-west-2

howdoicomputer commented 9 months ago

Setting environment variables also doesn't work; the only way I've gotten cdktf to read in my secret keys is by specifying them in my code and ehhhhhhhhh

aleksclark commented 3 weeks ago

experiencing the same issue - tf and awscli use my credentials fine, cdktf does not

aleksclark commented 3 weeks ago

man no sooner do a I comment.... @howdoicomputer I resolved this by specifying AWS_PROFILE=dev when calling cdktf (matching my configured profile) - maybe specifying default would resolve?

hashicorp / terraform-cdk