AWS CLI Access Terraform CA module init

[smallinsky]

What

When terraform module is initialized from S3 bucket using Teleport tsh proxy aws

module "mymodule" {
    source  = "s3::https://***.s3.eu-central-1.amazonaws.com/mymodule/1.0.0.zip"
}

The module http flow doesn't respect the AWS_CA_BUNDLE env var. In the result the following error is returned:

terraform init
2023-06-19T14:57:41.849+0200 [INFO]  Terraform version: 1.3.6
2023-06-19T14:57:41.849+0200 [INFO]  Go runtime version: go1.19.3
2023-06-19T14:57:41.849+0200 [INFO]  CLI args: []string{"terraform", "init"}
2023-06-19T14:57:41.850+0200 [INFO]  CLI command args: []string{"init"}
Initializing modules...
Downloading s3::https://test-bucket1.s3.eu-central-1.amazonaws.com/mymodule/1.0.0.zip for mymodule...
╷
│ Error: Failed to download module
│
│ Could not download module "mymodule" (main.tf:1) source code from "s3::https://test-bucket1.s3.eu-central-1.amazonaws.com/mymodule/1.0.0.zip": RequestError: send request failed
│ caused by: Get "https://bucket1.s3.eu-central-1.amazonaws.com/mymodule/1.0.0.zip": x509: “test-bucket1.s3.eu-central-1.amazonaws.com” certificate is not trusted

Related: https://github.com/hashicorp/terraform/issues/28551

Workaround

1) Create a aws profile with local teleport CA:

$ cat ~/.aws/config
[profile aws-app]
ca_bundle = /Users/marek/.tsh/keys/ice-berg.dev/marek-app/ice-berg.dev/awsconsole-localca.pem

2) Start aws local proxy

tsh proxy aws --app awsconsole -p 5555
Started AWS proxy on http://127.0.0.1:5555.

Use the following credentials and HTTPS proxy setting to connect to the proxy:
  export AWS_ACCESS_KEY_ID=aws-key-id-value
  export AWS_SECRET_ACCESS_KEY=aws-secret-access-key
  export AWS_CA_BUNDLE=/Users/marek/.tsh/keys/ice-berg.dev/marek-app/ice-berg.dev/awsconsole-localca.pem
  export HTTPS_PROXY=http://127.0.0.1:5555

3) Export local proxy env:

  export AWS_ACCESS_KEY_ID=aws-key-id-value
  export AWS_SECRET_ACCESS_KEY=aws-secret-access-key
  export AWS_CA_BUNDLE=/Users/marek/.tsh/keys/ice-berg.dev/marek-app/ice-berg.dev/awsconsole-localca.pem

Export AWS_SDK_LOAD_CONFIG and AWS_PROFILE created in step 1.

export AWS_SDK_LOAD_CONFIG=1
export AWS_PROFILE=app-aws

4) Run terraform script:

$ terraform init

Initializing the backend...
Initializing modules...
Downloading s3::https://marek-test-bucket1.s3.eu-central-1.amazonaws.com/mymodule/1.0.0.zip for mymodule...
- mymodule in .terraform/modules/mymodule

Initializing provider plugins...

Terraform has been successfully initialized!

[Tener]

@smallinsky

I'm looking into this, but it looks like we do have --exec flag:

$ tsh --help aws
usage: tsh aws [<flags>] [<command>...]

Access AWS API.

Flags:
...
      --exec                     Execute different commands (e.g., terraform) under Teleport credentials

This gives you correct randomly changing credentials on each run:

$ tsh aws --exec sh -- -c export | grep -i aws
export AWS_ACCESS_KEY_ID="9872bafd-353a-4f06-8f76-d4c6d28e63e6"
export AWS_CA_BUNDLE="/Users/tener/.tsh/keys/boson.tener.io/tener-app/boson.tener.io/awsconsole-prod-localca.pem"
export AWS_SECRET_ACCESS_KEY="1ed2ee08-1619-4993-8841-dee6385af7a6"

$ tsh aws --exec sh -- -c export | grep -i aws
export AWS_ACCESS_KEY_ID="d2d6e00f-dbb2-47d6-a445-6a99f4336a2d"
export AWS_CA_BUNDLE="/Users/tener/.tsh/keys/boson.tener.io/tener-app/boson.tener.io/awsconsole-prod-localca.pem"
export AWS_SECRET_ACCESS_KEY="3e224feb-753a-45a2-a706-a224fafcea2a"

So you can simply do: tsh aws --exec terraform init.

Your thoughts?

[smallinsky]

So you can simply do: tsh aws --exec terraform init.

@Tener Have you tested this flow ? I don't think that it will work since the issue is related to AWS_CA_BUNDLE env not respected by one of internal library that is used in terraform codebase to fetch module dependency from aws s3 source.

In terms of technical flow there is not difference between tsh proxy aws --app and tsh aws --exec so these commands should give the same result.

[Tener]

All right, I was confused by the workaround step no. 3 exporting the AWS_CA_BUNDLE. Thanks for this.

[Tener]

Looking at the Terraform code, the fetch in module init is done with Hashicorp's go-getter library. Unfortunately, there is no direct support for AWS_CA_BUNDLE in that library.

Reading the S3Getter code points towards another possible workaround: specify the AWS profile using aws_profile URL param.

module "mymodule" {
    source  = "s3::https://***.s3.eu-central-1.amazonaws.com/mymodule/1.0.0.zip?aws_profile=app-aws"
}

Assuming you are fine with changing the .tf files to override the aws_profile param, this gives you a basis for a stable workaround.

Unfortunately, I don't see a way to make this work without user intervention. I'm not sure if there is a point in changing tsh behavior, as users would have to adapt their .tf files anyway.

I have raised PR to add support for AWS_CA_BUNDLE in go-getter, but I'm not sure if/when it is going to be merged: https://github.com/hashicorp/go-getter/pull/457.

Nevertheless, you can easily rebuild Terraform with my change:

git clone https://github.com/hashicorp/terraform.git terraform-git
cd terraform-git
go mod edit -replace github.com/hashicorp/go-getter=github.com/Tener/go-getter@b21a9339890e6c2a20c4dd638763de8755c7c665
go mod tidy
go install .

Given custom terraform installed to GOPATH, this will work:

tsh aws exec -- `go env GOPATH`/bin/terraform init

[smallinsky]

Assuming you are fine with changing the .tf files to override the aws_profile param, this gives you a basis for a stable workaround.

Unfortunately, I don't see a way to make this work without user intervention.

The go-getter seems to support the ca_bundle from AWS profile configuration. I was thinking about creating a teleport profile file in tsh directory during tsh app login aws-console-app call.

Does that sound reasonable ?

[Tener]

The go-getter seems to support the ca_bundle from AWS profile configuration. I was thinking about creating a teleport profile file in tsh directory during tsh app login aws-console-app call.

We could prepare this profile, but that is a heavy change. Currently, we use env variables to pass the config, including secrets. It isn't immediately clear what we should put in the profile - only the CA bundle or all of the other settings, too? There are arguments to be made either way.

The bigger problem is that even with the profile in place, we have no way to make terraform init use it on our own. The two mechanisms available are:

1. The special URL param: ?aws_profile=app-aws
2. Combination of AWS_SDK_LOAD_CONFIG and other env variables, e.g. AWS_PROFILE.

There is no way we should be modifying .tf files, so the first one is out. I don't think we should use the second either, as AWS_SDK_LOAD_CONFIG is a heavy hammer to wield and is likely to cause numerous side issues.