Running terraform in a docker container against aws results in failure to authenticate

[ghost]

Using a simple docker file like this:

FROM alpine

ADD ./bin/*   /bin/

VOLUME ["/workspace"]
WORKDIR "/workspace"

ENTRYPOINT ["/bin/terraform"]
CMD []

Building it with a script similar to:

VERSION=0.6.11
UNIT=${UNIT:-terraform}
UNIT_URL=${UNIT_URL:-https://releases.hashicorp.com}
LATEST=${LATEST:-false}
if [ ${LATEST} = "true" ];
then
  VERSION=$(curl -s $UNIT_URL/$UNIT/ | grep -E "$UNIT.[0-9]{1,3}" | head -1 | sed -e 's^\(<a href=\|</a>\|"\| \)^^g' -e 's/>/:/' -e 's^</a>^^' | awk -F: '{print$NF}' | awk -F_ '{print$NF}')
fi
REGISTRY=${REGISTRY:-aidevops}
ARGS=${ARGS:---pull}
TAG=${TAG:-latest}
...
echo "=============================================================================================="
echo "Downloading $UNIT $VERSION from: $UNIT_URL/${UNIT}/${VERSION}/${UNIT}_${VERSION}_linux_amd64.zip"
echo "=============================================================================================="

mkdir -p bin
if [ "`uname`" == "Darwin" ];
then
  curl -s -L $UNIT_URL/${UNIT}/${VERSION}/${UNIT}_${VERSION}_linux_amd64.zip | tar -xf - -C bin
else
  curl -s -L $UNIT_URL/${UNIT}/${VERSION}/${UNIT}_${VERSION}_linux_amd64.zip -o /tmp/${UNIT}_${VERSION}_linux_amd64.zip && \
    unzip -o -d ./bin /tmp/${UNIT}_${VERSION}_linux_amd64.zip
fi

echo "=============================================================================================="
echo "Building \`$REGISTRY/$UNIT:$TAG\`"
echo "=============================================================================================="

docker build $ARGS -t $REGISTRY/$UNIT:$TAG .
...

and running it with a call similar to:

docker run --rm -it -e PATH=/bin:/usr/bin:/usr/local/bin \
  -e CONSUL_HTTP_TOKEN=$CONSUL_HTTP_TOKEN \
  -e TF_VAR_terraform_home=$SUSHI_REMOTE_HOME/foo \
  -e TF_VAR_aws_access_key=$TF_VAR_aws_access_key \
  -e TF_VAR_aws_secret_key=$TF_VAR_aws_secret_key \
  -v $SUSHI_REMOTE_HOME:$SUSHI_REMOTE_HOME \
  -v $LOC:/workspace \
  -v $LOC:$LOC \
  -v $SUSHI_REMOTE_HOME/.terraform.d:/root/.terraform.d \
  --workdir=$LOC --entrypoint=/bin/${UNIT} $REGISTRY/$UNIT:$TAG ${@: 1}

or interactively in-container

docker run --rm -it -e PATH=/bin:/usr/bin:/usr/local/bin \
  -e CONSUL_HTTP_TOKEN=$CONSUL_HTTP_TOKEN \
  -e TF_VAR_terraform_home=$SUSHI_REMOTE_HOME/foo \
  -e TF_VAR_aws_access_key=$TF_VAR_aws_access_key \
  -e TF_VAR_aws_secret_key=$TF_VAR_aws_secret_key \
  -v $SUSHI_REMOTE_HOME:$SUSHI_REMOTE_HOME \
  -v $LOC:/workspace \
  -v $LOC:$LOC \
  -v $SUSHI_REMOTE_HOME/.terraform.d:/root/.terraform.d \
  --workdir=$LOC --entrypoint=/bin/sh $REGISTRY/$UNIT:$TAG ${@: 1}

followed by:

size=3 etcd_token=$(curl -s https://discovery.etcd.io/new?size=$size) terraform apply --var "etcd_token=${etcd_token}" --var SUSHI_HOME=$SUSHI_HOME --var-file=./dev.tfvars --var-file=$SUSHI_HOME/tf_secret.tfvars

Yields the error:

* Error loading credentials for AWS Provider: NoCredentialProviders: no valid providers in chain

Other useful info:

Terraform version

Terraform v0.6.11

Docker version

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.1
 Git commit:   a34a1d5
 Built:        Sat Nov 21 00:48:57 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   4419fdb-dirty
 Built:        Fri Dec 18 00:01:03 UTC 2015
 OS/Arch:      linux/amd64

Docker client uname

Darwin projim.local 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64

Docker host version

Linux docker.vm 4.3.3-coreos #2 SMP Thu Dec 17 23:57:55 UTC 2015 x86_64 Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz GenuineIntel GNU/Linux

[ghost]

While running in container the following environment is available:

-> % ~/containers/dockerfiles/swiss-sushi-knife/bin/terraform                                                                                                                                                                                                                                                             [22:08:58]  [containers *] 48e3741
WARNING: login credentials saved in /workspace/.docker/config.json
Login Succeeded

/workspace/deployments/core-containers-dev # env
TF_VAR_aws_access_key=XXXXXXXXXX
HOSTNAME=247c5ea2a05b
SHLVL=1
HOME=/root
TF_VAR_encrypted_data_bag_secret=/workspace/.chef/encrypted_data_bag_secret
TF_VAR_chef_validation_key=/workspace/.chef/validator.pem
CONSUL_HTTP_TOKEN=XXXXXXXX-XXXX-XXXX-e6cc-a4844063cd78
TERM=xterm
SUSHI_REMOTE_HOME=/workspace
PATH=/bin:/usr/bin:/usr/local/bin
TF_VAR_aws_secret_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXi
TF_VAR_terraform_home=/workspace
PWD=/workspace/deployments/core-containers-dev
/workspace/deployments/core-containers-dev # size=3 etcd_token=$(curl -s https://discovery.etcd.io/new?size=$size) terraform apply --var "etcd_token=${etcd_token}" --var SUSHI_HOME=$SUSHI_HOME --var-file=./dev.tfvars --var-file=$SUSHI_REMOTE_HOME/tf_secret.tfvars
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Default: us-east-1
  Enter a value:

template_file.user_data: Refreshing state... (ID: 194e45c38c703663033c841f0006b3a189b04a2de4e813970bcad75ae22f4f71)
Error refreshing state: 1 error(s) occurred:

* 1 error(s) occurred:

* No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider
/workspace/deployments/core-containers-dev # ls -l /bin/terraform*
-rwxr-xr-x    5 root     root      26557064 Feb  1 19:17 /bin/terraform
-rwxr-xr-x    5 root     root      14952136 Feb  1 19:17 /bin/terraform-provider-atlas
-rwxr-xr-x    5 root     root      42663384 Feb  1 19:20 /bin/terraform-provider-aws
-rwxr-xr-x    5 root     root      17985816 Feb  1 19:19 /bin/terraform-provider-azure
-rwxr-xr-x    5 root     root      23355368 Feb  1 19:19 /bin/terraform-provider-azurerm
-rwxr-xr-x    5 root     root      15061280 Feb  1 19:20 /bin/terraform-provider-chef
-rwxr-xr-x    5 root     root      14837280 Feb  1 19:20 /bin/terraform-provider-cloudflare
-rwxr-xr-x    5 root     root      26391560 Feb  1 19:17 /bin/terraform-provider-cloudstack
-rwxr-xr-x    5 root     root      15221056 Feb  1 19:17 /bin/terraform-provider-consul
-rwxr-xr-x    5 root     root      15527576 Feb  1 19:17 /bin/terraform-provider-digitalocean
-rwxr-xr-x    5 root     root      14858680 Feb  1 19:17 /bin/terraform-provider-dme
-rwxr-xr-x    5 root     root      14848160 Feb  1 19:17 /bin/terraform-provider-dnsimple
-rwxr-xr-x    5 root     root      16036816 Feb  1 19:17 /bin/terraform-provider-docker
-rwxr-xr-x    5 root     root      14862512 Feb  1 19:17 /bin/terraform-provider-dyn
-rwxr-xr-x    5 root     root      22236904 Feb  1 19:17 /bin/terraform-provider-google
-rwxr-xr-x    5 root     root      15500448 Feb  1 19:17 /bin/terraform-provider-heroku
-rwxr-xr-x    5 root     root      14857328 Feb  1 19:18 /bin/terraform-provider-mailgun
-rwxr-xr-x    5 root     root      15487232 Feb  1 19:17 /bin/terraform-provider-mysql
-rwxr-xr-x    5 root     root      14749136 Feb  1 19:17 /bin/terraform-provider-null
-rwxr-xr-x    5 root     root      17871288 Feb  1 19:17 /bin/terraform-provider-openstack
-rwxr-xr-x    5 root     root      15174248 Feb  1 19:17 /bin/terraform-provider-packet
-rwxr-xr-x    5 root     root      15396488 Feb  1 19:17 /bin/terraform-provider-postgresql
-rwxr-xr-x    5 root     root      14827704 Feb  1 19:17 /bin/terraform-provider-powerdns
-rwxr-xr-x    5 root     root      15124360 Feb  1 19:17 /bin/terraform-provider-rundeck
-rwxr-xr-x    5 root     root      14865032 Feb  1 19:17 /bin/terraform-provider-statuscake
-rwxr-xr-x    5 root     root      14842032 Feb  1 19:20 /bin/terraform-provider-template
-rwxr-xr-x    5 root     root      23343320 Feb  1 19:20 /bin/terraform-provider-terraform
-rwxr-xr-x    5 root     root      15170568 Feb  1 19:20 /bin/terraform-provider-tls
-rwxr-xr-x    5 root     root      15508952 Feb  1 19:20 /bin/terraform-provider-vcd
-rwxr-xr-x    5 root     root      44742856 Feb  1 19:21 /bin/terraform-provider-vsphere
-rwxr-xr-x    5 root     root      16248320 Feb  1 19:20 /bin/terraform-provisioner-chef
-rwxr-xr-x    5 root     root      16208776 Feb  1 19:20 /bin/terraform-provisioner-file
-rwxr-xr-x    5 root     root      14348376 Feb  1 19:20 /bin/terraform-provisioner-local-exec
-rwxr-xr-x    5 root     root      16191808 Feb  1 19:20 /bin/terraform-provisioner-remote-exec
/workspace/deployments/core-containers-dev # /bin/terraform-provider-aws
This binary is a Terraform plugin. These are not meant to be
executed directly. Please execute `terraform`, which will load
any plugins automatically.
/workspace/deployments/core-containers-dev #

[ghost]

Use this dockerfile instead if you don't want to have to manually install curl for the example above.

FROM alpine

ADD ./bin/*   /bin/

RUN \
  apk update && apk add curl && rm -rf /var/cache/apk/*

VOLUME ["/workspace"]
WORKDIR "/workspace"

ENTRYPOINT ["/bin/terraform"]
CMD []

[ghost]

Amended run command

Added

docker run --rm -it --privileged=true -e HOME=/workspace ...

still no dice..

[ghost]

I would like to containerize this as part of our CI/CD process, is that supported?

[phinze]

Hi @johnt337 - definitely odd behavior you're seeing here!

The env vars you're providing assume that there is config like this somewhere to wire the Terraform variables into the provider:

variable "aws_access_key" {}`
variable "aws_secret_key" {}`

provider "aws" {
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
}

Another option is to drop that config style and go directly to provider variables. Do you get different behavior if you switch to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?

Terraform should definitely be able to run in a Docker environment - we can get this sorted out! :+1:

[ghost]

Did you want that prefixed with TFVAR?

[phinze]

@johnt337 TF_VAR_foo sets a the value for a Terraform Variable foo, which would be declared in config like:

variable "foo" {}

I'm suggesting giving the AWS Provider specific environment variables a shot - check out "Environment Variables" under the AWS Provider docs. That doc also details the other methods for configuring the AWS provider.

[ghost]

I'm going to try it as you have it listed, per https://www.terraform.io/docs/providers/aws/.

[ghost]

Ah! Our messages just crossed, OK thanks!

[ghost]

That looks to have worked! Is there a way to take an environment variable and pass it into a resource?

Similar to how I was doing that here?

# Template file for GETTING instance user data file
resource "template_file" "user_data" {
  template = "./templates/config.tpl"
  vars {
    version = "${lookup(var.deployment, "version")}"
    environment = "${lookup(var.deployment,"environment")}"
    app_name = "${lookup(var.deployment,"app_name")}"
    region = "${lookup(var.deployment,"region")}"
    ssh_key     = "${lookup(var.auth,format("%s.ssh_key",lookup(var.deployment,"ssh_key")))}"
    ssh_key_name= "${lookup(var.deployment,"ssh_key")}"
    customer = "${lookup(var.deployment,"customer")}"
    consortium = "${lookup(var.deployment,"consortium")}"
    consul_url  = "${lookup(var.deployment,"consul_url")}"
    consul_dc   = "${lookup(var.deployment,"consul_dc")}"
    dns_domain  = "${lookup(var.deployment,"dns_domain")}"
    os  = "${lookup(var.deployment,"os")}"
    os_version  = "${lookup(var.deployment,"os_version")}"
    instance_size  = "${lookup(var.deployment,"instance_size")}"
    cluster_size = "${lookup(var.deployment,"cluster_size")}"
    role = "${format("%s-%s", lookup(var.deployment,"app_name"), lookup(var.deployment,"role_name"))}"
    node_name   = "${format("%s-%s-%s", lookup(var.deployment,"app_name"), lookup(var.deployment,"role_name"), lookup(var.deployment,"environment"))}"
    aws_tag_version = "${lookup(var.deployment,"aws_tag_version")}"
    etcd_discovery = "${var.etcd_token}"
    aws_registry_id    = "${lookup(var.deployment,"aws_registry_id")}"

    /*
    The following vars are sourced from /etc/aws_tag_instance.sh
      name           - combines <role>-<instance_id>.<env>
      instance_id    - running instance_id via 169.254.169.254/latest/meta-data
      ami_id         - instantianted ami_id via 169.254.169.254/latest/meta-data
    */

    /*
     these vars are sourced via env TF_VAR_... and written to
     /etc/aws_environment temporarily until tagged; then its removed.
    */
    # aws_access_key = "${var.aws_access_key}"
    # aws_secret_key = "${var.aws_secret_key}"
  }
}

[ghost]

Also does this work with IAM profiles and temporary tokens?

[ghost]

In the short term, i'm trying to use IAM profiles to get around the config based key entries.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.