Allow terraform init to define backend type as a flag

[reegnz]

Current Terraform Version

Terraform v0.12.24

Use-cases

I am using the -backend-config flag to pass partial backend configuration to terraform init. I would also like declare the backend type as a flag, eg. -backend-type.

I would really like to use terraform like this:

export TF_CLI_ARGS_init="-backend-type=s3 -backend-config=dev-backend.tfvars -from-module=git::https://example.com/modules.git//amodule?ref=1.2.3"
export TF_CLI_ARGS_apply="-var-file dev.tfvars"

terraform init
terraform apply 

This way I can reuse the same module across multiple environments without duplicating any tf files, and also declare different backend types for different environments as well (eg. some in s3, some in terraform enterprise). This means my environments, including backend types and config can be managed from tfvars files, reusing the same module code everywhere.

Attempted Solutions

Right now to achieve the above use-case, I need to define a new module that wraps the actual desired module, re-exposes the module outputs, declares proxy-variables, somewhat like this:

terraform {
  backend "s3" {}
}

module "amodule" {
  source = "git::https://example.com/modules.git//amodule?ref=1.2.3"
  var1 = var.var1
  var2 = var.var2
}

variable "var1" {
}
variable "var2" {
}
output "out1" {
  value = module.amodule.out1
}
output "out2" {
  value = module.amodule.out2
}

This kind of wrapping also means I have to maintain this sort-of proxy-module that only purpose is to wrap another module to add a backend configuration. This kind of code is something I feel is very verbose given how small of a feature it actually provides.

Proposal

Add a new flag to terraform init called -backend-type that allows defining or overriding the backend type for the module.

References

• 19300

[seanorama]

Similar simpler use case:

My backend configs are abstracted for each environment with -backend-config=${env}/backend.conf except for the need for a .tf file with:

terraform {
  backend "s3" {}
}

As we have different backend types across our code base, this makes it much harder to abstract and causes a lot of duplication.

This would be solved by having -backend-type.

[reegnz]

I checked the code for on the terraform 0.12 branch to see if this is an easy change.

For init it's an easy change, managed to create a working solution.

Problem is, the rest of the commands (apply, plan, destroy) will whine about an initialized remote backend if there's no .tf config present for it. It recognizes falsely that the backend configuration has changed, and wants to migrate the backend to local config:

https://github.com/hashicorp/terraform/blob/7c646ddeced3c30517615c1ed54a4a5802726167/command/meta_backend.go#L494-L504

Now this could only be fixed if we either don't assume backend has to be reconfigured if there's no backend config in the .tf files, and just go with the one the module got initialized with (eg. delete the logic for reconfiguring from remote to local).

[apparentlymart]

Another option to do this today is to follow how Terraform Cloud sets up the backend for a particular run. Before running terraform init, Terraform Cloud generates a .tf.json file containing something like this:

{
  "backend": {
    "remote": {
      "hostname": "app.terraform.io",
      "organization": "example",
      "workspaces": {
        "name": "example-workspace"
      }
    }
  }
}

Because there's nothing in this file that Terraform Cloud can't reproduce on the next run, this file is thrown away immediately after the run is complete. This allows Terraform Cloud to set all of the necessary backend settings at once, without packing them all into the terraform init command line.

(Terraform Cloud specifically generates an override file so that it can work with a configuration that already contains a backend block, but that isn't necessary if you know that none of your modules are going to have backend blocks inside already anyway.)

[reegnz]

@apparentlymart so how would you use -from-module with override files? The problem I ran into with -from-module is that it refuses to init to a non-empty dir, but if I don't have the override file in there, then init won't use the backend config, because I would have to put that in there after terraform init -from-module.

Maybe an option would be to allow terraform init -from-module not just into empty directories, but also directories already containing override files? Then init would work great without having to replace terraform init with a custom git clone + cp + terraform init wrapper.

EDIT: to clarify, I am trying to understand how I could actually use the -from-module flag, because I cannot find it's actual useful use-case.

[reegnz]

Another option could be to accompany -from-module with an -override flag as well, so that after fetching the module source, the override files would be copied to the working directory. Then existing override filenames could be explicitly overridden (eg. override file provided in the flag wins over module code).

To clarify the use-case:

• You have a module with no backend config (eg. in git, or terraform module registry)
• You fetch the module with -from-module
• You want to configure a backend (type + config) for it during init
• Yo may want to override provider configs as well during init (if they existed in the upstream module)

The more I think about it, the latter two definitely seem more of a use-case for override files.

I will have a look at the code to see how difficult adding an '-override' flag to init would be (and luckily the implementation definitely would only affect init and not other commands).

[apparentlymart]

The terraform init -from-module option is there primarily to enable easily using modules from the Terraform Registry as an example starting point for your own local fork. The idea was that you'd then make local modifications to the code (such as adding a backend block) and then push the result to your own VCS repository, and then you'd use the normal Terraform workflow (without -from-module) on that local repository moving forward.

For example:

terraform init -from-module=hashicorp/consul/aws

-from-module is not something that was intended for everyday use. Indeed, its mostly vestigial at this point, because there was in the past a plan to include some instructions in the Terraform Registry UI that used it, but the UI there now uses an example module block instead, because that's the more common way to make use of a module. (In practice, most modules in the registry are designed to be used as child modules, not as root modules.)

The current intended way to use Terraform is to put your backend configuration in your version control repository in the root module. Override files are an alternative if you are providing the backend configuration mechanically using some surrounding automation. In the case where you are automating Terraform and generating the backend configuration, I'd suggest the following steps:

• Clone the repository containing your configuration from version control into a local directory.
• Inside that directory, generate an _override.tf.json file containing the backend configuration.
• Run terraform init to prepare the working directory for use, including activating the backend from the generated file.
• Run terraform plan and terraform apply as described in Running Terraform in Automation.

[reegnz]

Yupp, my plan for doing it right now is to write a wrapper that:

1. does a git clone of the module
2. puts an override file containing the backend config
3. init + plan + apply

I would have preferred not having to wrap terraform for such a use-case, but for the tool to provide this out of the box.

How would I use that approach you just described in terraform cloud?

To be exact, I am trying to reduce boilerplate code here, see my opening comment under 'attempted solutions'. How would I do that by not using -from-module?

Do I have to choose between writing that boilerplate, writing my own wrapper, or using a 3rd party wrapper like terragrunt? I'd rather avoid the latter two and use vanilla terraform (since the latter two are not supported by terraform cloud).

[apparentlymart]

This particular situation is not really relevant in Terraform Cloud (assuming you mean when using remote operations) because runs in Terraform Cloud always use the remote backend, and Terraform Cloud already does those three steps you described automatically in order to configure the remote backend.

To be clear, I'm not leaving these comments for any reason other than to help folks who have these requirements today and want to solve them with Terraform as it currently exists. That doesn't mean Terraform can't be changed, but when there's already a solution available I think it's helpful to describe that solution to avoid folks feeling like they have to block waiting for an issue to be closed before solving their problem.

[reegnz]

So now that terraform 0.15 has gotten a bunch of useful features, could we re-start this discussion please?

How I wish to use terraform is, without having to change module code:

1. clone a terraform module repo that doesn't have any backend config or tfvars defined in it (it's a module, duh!)
2. Using terraform flags or env variables configure a backend and tfvars
3. init, plan, apply

For init:

1. define a data dir: TF_DATA_DIR
2. define backend type -> this is the one still missing!
3. define backend config: achievable with --backend-config
4. use -chdir to switch to the module dir

For plan/apply:

1. define tfvars: achievable with a multitude of options, -var, -var-file, TF_VAR_var_name.
2. define data dir: TF_DATA_DIR
3. use -chdir

So could we get a -backend-type flag to fit that little hole in the init process?

If I prepare a PR for that, what are the chances of it being merged? I'm open to contribute, but don't want to put any effort into it if I'm not seeing any openness for this. :)

[fv-ian]

I would be very interested in this.

I have a workflow that we wish to support better which is a local backend is the default configuration for people to test and develop against. This is fast turnaround for purposes of testing against say their own development AWS Account.

Then when they have something working mostly right, they push that into version control which would then run a pipeline that would attach the backend-config to the pipeline version and store the tfstate into a central repository location that is highly locked down and controlled.

Example:

Local Development: -> Defaults to local backend, all other configs are either variables passed in (tfvars or cli) or use the systems default patterns of say creds in ~/.aws/credentials in the [default] profile

Pipeline Prod Deployments; -> configured to use s3 backend that is tightly controlled to only be accessed via pipeline using terraform workspaces and appropriate reviews with approvals. The configs are passed in via a tfvars file that is configured dynamically depending on the environment being deployed to.

[schollii]

I have taken a completely different approach to the backend problem: I created a module that is entirely for storing any number of tf state backends. Create it once, never worry about it again, because the module creates the backend.tf files itself. It also makes it trivial to split off a root module into other root modules (that share state via remote states). It's awesome, I use it in several projects, but it is still early stage and currently focussed on s3. I know it's not for everyone; some are averse to having more than one state per bucket but I find that does not scale. The code could surely be extended to support other backends. See https://registry.terraform.io/modules/schollii/multi-stack-backends/aws for details.

[fv-ian]

@schollii This doesn't address the problem being raised here. How is this different than using terraform workspaces which solves a different challenge?

[timini]

how to do it?

[schollii]

@fv-ian you create tfstate backend resources once for the account (s3, dyndb table, iam policies etc), then all cicd has to do at commit time is create the backend.tf (or .hcl if using the -backend-config init option).

[LanceNero]

Add a scenario:

Ansible Direct Call Terraform Module

Scenario Background

We defined many terraform module on locally, when we need to deploy a new infras via terraform, we'll using ansible call terraform module(the module will call terraform cmd), so that mean we'll not create another main.tf and define a module block in main.tf.

Directory structure

.
├── resources
│   ├── terraform-modules
│   │   ├── create-vpc
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       ├── README.md
│   │       ├── variables.tf
│   │       └── versions.tf
└── site.yml

main.tf

## We'll not define a backend block here for the concept of decoupling

resource "null_resource" "test" {
    provisioner "local-exec" {
        command = "echo just a test"
    }

}

site.yml

---
- name: "[ Part 1 ]Creating Infrastructure...."
  hosts: localhost
  connection : local
  gather_facts: no 
  vars:
    project_path: XXXXX
    custom_path: XXXXXXXXX

tasks:
    - name: "Creating VPC - Planing"
      terraform:
        backend_config:
         ### didn't work, because we're not defined an empty local backend block in main.tf
          path: ./test.tfstate
        project_path: "{{ project_path }}"
        state: present
        force_init: true
        overwrite_init: false
        workspace: default
      register: present_result

Result

If we can add a way to dynamically define the backend type (such as above suggestion), that will helpful when users direct call the terraform module.

BTW, I've tried generate a .tf.json / .tf or _overwrite.tf.json file to the terraform running dir, but didn't work, seems like it caused I didn't define an empty 'local' backend in main.tf

.tf.json

{
  "terraform": {
    "backend": {
      "local": {
        "path": "./test.tfstate"
      }
    }
  }
}

[voltagex]

Adding my "vote" to this - I'd find it very useful for switching backends on the fly, for integration tests or local development.

At the moment the way backend overriding and -backend-config flags works can be quite confusing. For the moment I have worked around this by using a Makefile which renames backend.tf as needed.

[silopolis]

Adding my vote to this as a solution for the chicken-and-egg problem of provisioning the backend for a configuration in itself.

Configuration could hold the final config and the backend could be provisioned with tf apply -backen-type=local -target=s3_backend, then another round of bare init and apply and you're good to go.

This is somehow close to @fv-ian use case.