Terraform init on CodeBuild "no valid credential sources for S3 Backend found."

marian-gheorghe commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform 1.2.7 AWS Provider 3.75.2

Affected Resource(s)

S3 Backend Config

Terraform Configuration Files

I have AWS CodeBuild which is assuming role devops

codebuild.tf

resource "aws_codebuild_project" "code_build" {
  name          = "${var.app_name}-${var.target_env}-${var.build_project}"
  description   = "${var.app_name} ${var.build_project} pipeline on ${var.target_env}"
  service_role  = arn:aws:iam::xxxxx:role/devops
....

devops role has a policy that allows to be assumed by codebuild service and comes with admin privileges

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Principal": {
                    "Service": "codebuild.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }

Admin privileges

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "*",
                "Resource": "*"
            }
        ]
    }

Terraform backend config

    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~>3.0"
        }
      }

      backend "s3" {
        bucket         = "tfstatebucket"
        key            = "infrastructure/terraform.tfstate"
        region         = "eu-central-1"
        dynamodb_table = "cdc-terraform-up-and-running-lock"
        encrypt        = true
      }
    }

    provider "aws" {
      region = lookup(var.aws_region, var.env)

      allowed_account_ids = [
        lookup(var.account_id, var.env),
      ]
    }

Ohter configs

Buildspec.yaml

    version: 0.2

    phases:
      install:
        commands:
          - |
            if [ -n "$INSTALL_TOOLS_SCRIPT" ]; then
              ./$INSTALL_TOOLS_SCRIPT
            fi
      pre_build:
        commands:
          - |
            if [ -n "$PREBUILD_SCRIPT" ]; then
              ./$PREBUILD_SCRIPT
            fi
      build:
        commands:
          - |
            if [ -n "$BUILD_SCRIPT" ]; then
              ./$BUILD_SCRIPT
            fi
      post_build:
        commands:
          - |
            if [ -n "$POSTBUILD_SCRIPT" ]; then
              ./$POSTBUILD_SCRIPT
            fi

install_tools.sh

    #!/bin/bash
    set -euxo pipefail

    function install_terraform() {
        echo Installing Terraform...

        curl -s -qL -o terraform_install.zip https://releases.hashicorp.com/terraform/1.2.7/terraform_1.2.7_linux_amd64.zip
        unzip terraform_install.zip -d /usr/bin/
        chmod +x /usr/bin/terraform

        terraform --version 
    }

    install_terraform

prebuild.sh

    set -euxo pipefail

    echo Terraform init...
    terraform init

Expected Behavior

terraform init should succeed

Actual Behavior

Terraform init...
+ terraform init
Initializing modules...
- aws_appautoscaling_ecs_consumer_target in tfmodules/autoscaling
- aws_appautoscaling_ecs_server_target in tfmodules/autoscaling
- aws_appautoscaling_ecs_websocket_server_target in tfmodules/autoscaling
 
Initializing the backend...

 Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.

 Please see https://www.terraform.io/docs/language/settings/backends/s3.html
 for more information about providing credentials.

 Error: NoCredentialProviders: no valid providers in chain. Deprecated.
   For verbose messaging see aws.Config.CredentialsChainVerboseErrors

DEBUG LOG

Debug Log Gist

Steps to Reproduce

terraform init

Important Factoids

It works fine on local machine after successfully assuming the devops role. It doesn't work on CodeBuild CodeBuild is configured in a private VPC (VPC with only private subnets)

References

0000

marian-gheorghe commented 2 years ago

For the sake of it I have manually attempted assume-role via cli successfully, parsed the credentials and exported corresponding AWS env vars. However, terraform init still fails, complaining about wrong session token. How is that even possible?

Log Output

+ aws sts assume-role --role-arn arn:aws:iam::***:role/devops --role-session-name codebuild
+ cat creds
{
    "Credentials": {
        "AccessKeyId": "***",
        "SecretAccessKey": "***",
        "SessionToken": "***",
        "Expiration": "2022-08-23T21:47:47+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "***:codebuild",
        "Arn": "arn:aws:sts::***:assumed-role/devops/codebuild"
    }
}
++ jq .Credentials.AccessKeyId
+ export 'AWS_ACCESS_KEY_ID="*"'
+ AWS_ACCESS_KEY_ID='"*"'

++ jq .Credentials.SecretAccessKey
+ export 'AWS_SECRET_ACCESS_KEY="*"'
+ AWS_SECRET_ACCESS_KEY='"*"'

++ jq .Credentials.SessionToken
+ export 'AWS_SESSION_TOKEN="*"'
+ AWS_SESSION_TOKEN='"*"'

+ terraform init
Initializing modules...
- aws_appautoscaling_ecs_consumer_target in tfmodules/autoscaling
- aws_appautoscaling_ecs_server_target in tfmodules/autoscaling
- aws_appautoscaling_ecs_websocket_server_target in tfmodules/autoscaling

Initializing the backend...
╷
│ Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
│   status code: 403, request id: 87aedbae-938c-4019-a82a-47a53dfe06f5
│

Associated code with the above output

aws sts assume-role --role-arn arn:aws:iam::***:role/devops --role-session-name codebuild > creds

cat creds

export AWS_ACCESS_KEY_ID=$(cat creds | jq '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(cat creds | jq '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(cat creds | jq '.Credentials.SessionToken')

TF DEBUG

marian-gheorghe commented 2 years ago

I've managed to find the mystery. Internally aws sdk is querying the credentials endpoint 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI for credentials. Because my CodeBuild instance was configured to run in private VPC, it had to go through corporate proxy for external resources. In no_proxy/NO_PROXY only the instance metadata IP (169.254.169.254) was whitelisted. Whitelisting 169.254.170.2 in my proxy configuration solved the problem.

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws