AWS credential environment variables not working for S3 backend

[jimcroft]

I was looking to use terragrunt for a new TF 0.12 project but on terragrunt init I get the following error output.

[terragrunt] 2019/10/09 17:18:03 Initializing remote state for the s3 backend
[terragrunt] 2019/10/09 17:18:03 Error initializing session: CredentialRequiresARNError: credential type source_pr
ofile requires role_arn, profile default
[terragrunt] 2019/10/09 17:18:03 Unable to determine underlying exit code, so Terragrunt will exit with error code 

My remote state config contains no AWS IAM creds or role details (just the same as our existing Terraform projects). If add the backend config into the target module I was intending to wrap terragrunt around, and run terraform init all's well.

The error looks exactly the same as another issue in another project I've seen recently. Ref: https://github.com/hashicorp/packer/issues/8036#issuecomment-532118566

Not sure if there's something odd going on with the Go AWS SDK here.

[yorinasub17]

Can you elaborate what your environment config is? How are you passing the credentials? The reason I ask is that it looks like it is trying to source the credentials from ~/.aws/config, and the error indicates that is misconfigured.

Also, it would help if you can share what other projects you have seen it working successfully with.

[jimcroft]

Sure thing.

We use multiple AWS accounts and enforce MFA on use of privileged IAM roles, even in dev.

To manage this we use the https://github.com/remind101/assume-role tool to switch between roles. This calls sts:AssumeRole, prompts for MFA token, then sets the various AWS_* environment variables used by the AWS CLI, and pretty much all other tools that follow the default precedence set by AWS SDKs (pretty much everything uses precedence set here https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html), with env vars being used if set.

We use Terraform, the AWS CLI, (non broken versions of) Packer, and a few custom tools written in Python/boto3, Go, etc. using this method.

Working terraform init in vanilla terraform tf file:

terraform {
  backend "s3" {
    bucket               = "our-bucket"
    key                  = "some-key"
    region               = "eu-west-1"
    dynamodb_table       = "terraform-lock"
    encrypt              = true
  }
}

Not working terragrunt init with backend config in terragrunt.hcl file instead:

remote_state {
  backend = "s3"
  config = {
    bucket                = "our-bucket"
    key                   = "some-key"
    region                = "eu-west-1"
    dynamodb_table        = "terraform-lock"
    encrypt               = true
  }
}

The terragrunt init command results in the error.

[jimcroft]

PS. So in summary our Terraform configs contain no credentials or auth details for the backend. They rely on the the following environment variables.

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SECURITY_TOKEN
AWS_SESSION_TOKEN

[yorinasub17]

• Do you have any entry in your aws config that has source_profile set without a role_arn?
• I use aws-vault, which is very similar to assume-role and have not had any problems with terragrunt reading those environment variables, so I am surprised it isn't reading it in your environment. Do you know if the tool is also setting AWS_PROFILE or do you have that set? I vaguely recall the go SDK using that to override the other env vars and start sourcing from the profile.

[jimcroft]

Yup that was it! I had a bogus and unused entry in ~/.aws/config that had a source_profile but no role arn, terragrunt init working now.

Thanks!

[reegnz]

I also ran into the same issue (config with source_profile but no role_arn), but aws cli still worked OK with that, only terragrunt failed. Shouldn't terragrunt (and actually all tools using aws-sdk) also work when the cli works? Principle of least surprise, eg. if cli works with the same env variables set, then code using the SDK should also just work.

[yorinasub17]

I agree with you in principle but practically speaking, there are situations that make it difficult to achieve full behavioral parity.

In general, all the SDKs honor the same properties for authentication, but they may drift depending on the version. E.g if there is a new feature that changes the parsing behavior of the config that was introduced in later versions of the SDK, terragrunt may fall behind because we statically link to a specific version of the SDK. Or you might be using an older version of the aws cli that has a bug that previously allowed relaxed constraints for the config that is now strictly checked in later versions and the SDK go matches that. (By the way, what version of the AWS CLI are you using?)

Another possibility to behavioral drift is if AWS itself has different build processes for the SDK such that one SDK flavor has drift in the feature. Note that the aws CLI implements many routines and functionalities that don't exist in the SDK, so I wouldn't be surprised if the AWS CLI is actually doing something advanced to allow a source_profile without role_arn that isn't available in other SDKs. In this case the issue should be filed on the aws-sdk-go project instead of here.

[reegnz]

The thing is, I am using aws-vault that uses aws-skd-go as well. Works without a problem. I use the 1.16.360 version of the cli currently, with the latest aws-vault.

So it might be an sdk issue, but I have a feeling that instead it's a problem with how the sdk is used.

And terragrunt is complaining because of a profile I'm not even using, it's just there in the config. And the error message isn't even user friendly, and it was pure chance I found this issue to help me (granted, that indeed is an sdk problem).

[yorinasub17]

Reopening the issue, but most likely won't get to this anytime soon. Happy to review a PR if you are able to figure out what we are doing wrong. The relevant code is here: https://github.com/gruntwork-io/terragrunt/blob/master/aws_helper/config.go#L27

[reegnz]

OK, I'll have a look, but can't guarantee anything. :)

EDIT: don't keep your hopes up, I stopped working with terragrunt. :(

[dmikalova]

Just ran into this issue. We have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set in environment variables, and then the backend is set to use a profile. In Terraform this just works. For the AWS CLI, if AWS_PROFILE is set then there needs to be a ~/.aws/config with that profile (even if just empty) for something like aws sts get-caller-identity to work. For Terragrunt it does not work at all, neither with the config file or the env var for the profile. It does work if we set everything in ~/.aws/config and ~/.aws/credentials explicitly.

[madhuryg]

I have the same issue as above. How did you resolve it ??

[jvguillen]

@dmikalova did you find a workaround? I'm having the same issue and clearing my ~/.aws/config and ~/.aws/credentials is not an option

[dmikalova]

@jvguillen I was doing this in a CI environment so I was able to automate writing the variables into the ~/.aws/ files.