Read backend configuration from source files

[mjnorman]

Hello all. So far loving the resource. However something I've noticed, and I'm not sure if I am missing something on this as far as best practices. I see that you specifically have to call out the s3 backend configuration in the source definition in the pipeline. Since this is not available to be overridden in the put step, it has to be known before hand. Usually this is configured and stored with the repo housing all of the terraform configuration as a backend.tf for instance. Is there a reason that it does not attempt to first read all configuration from the source first?

Since I have to both configure it in the repo so that I can terraform init and test locally, and in the pipeline, it goes against DRY. Am I missing something obvious on this?

Thanks for the work on the resource!

[ljfranklin]

The requirement of specifying the backend configuration in source is mostly due to Concourse's resource API. While the put step does have access to your git repo as an input, the check step can't take any inputs. Specifying the backend config directly in source means the check step can look for changes in statefiles without needing to access your git repo as input.

Implementation aside, generally speaking storing secrets in git repos feels like an anti-pattern anyway. Concourse has support for storing credentials in secrets managers like Vault: https://concourse-ci.org/vars.html. If you want to avoid duplication while still keeping your secrets in a git repo, you could do something like fly set-pipeline ... --var=backend_config="$(cat secrets/backend.tf)".

[mjnorman]

Oh I should have specified, not talking storing secrets, just the config for the backend. Like the Terraform example for s3 config below. Secrets would for sure be stored in Vault.

terraform {
  backend "s3" {
    bucket = "mybucket"
    key    = "path/to/my/key"
    region = "us-east-1"
  }
}

I was just surprised that it wouldn't read config from the source directly. It just seemed odd that it required duplication since nothing else seemed to require duplication of config.

[ljfranklin]

Ah, I see. Yeah, then the explanation I gave above is the reason: the put step does have access to your git repo as an input, the check step can't take any inputs. So the source block for the terraform-resource has to be self-contained and not depend on a git repo file.

[mjnorman]

Thanks!