Feature Request: Add flag for state files auto update.

[hdost]

Problem When a person uses a new version of Terraform it updates the state files by default.

Why is this a problem? The biggest reason for this it is currently very easy to overwrite your state and not have a backup, or even if you do have it there's not a great way to rollback. Having experienced this recently, if a single person on a team of people updates their version of Terraform including the person just starting with Terraform then it can cause problems with the files be overwritten with the new version.

Suggestion The addition of a flag --update-state=true would act as the current default behavior does today. Without the flag or with --update-state=false when using a newer version of terraform it would either warn that state cannot be updated OR the command returns an error code. Either way the state file should not be updated.

Alternatively, the old state should be preserved by making a backup copy of the file name with the version noted in the file name, and then the new state file retains the standard name and it performs the upgrade as expected. Along with that it might be worth adding an option --backup-state which defaults to true and could be set to --backup-state=false.

It might be a good idea that the same applies to Terraform Providers.

[mildwonkey]

Hi @hdost - I'm sorry you've run across this, and thanks for the feature request.

One way you can avoid this issue is specifying the required version of terraform in your configuration as described here:

terraform {
  required_version = "0.11.8 "
}

Provider versions can be specified as well.

[hdost]

We have actually done precisely that. I now need to verify that doing so prevents the state files from being modified.

[apparentlymart]

Hi @hdost,

As @mildwonkey described, we intend that required_version mechanism as the way for a team to "agree" on which version of Terraform they are all currently using and prevent the issue you're describing by requiring a configuration change (which would, in most organizations, go through code review) to opt in to an upgrade.

Since terraform apply must always update state in order to take any actions at all, the first option you proposed here would effectively disable terraform apply altogether, which I assume isn't really what you were looking for. terraform plan (with no arguments) is always safe to run with a newer version of Terraform in order to test what change an upgraded version of Terraform might make (e.g. as part of testing a change to that terraform_version setting) and will never update the state.

The usual workflow for a Terraform upgrade, then, is:

• Agree as a team that it's time to upgrade.
• Someone on the team downloads the new version you wish to upgrade to.
• That person changes required_version in their local dev environment to the new version.
• That person runs terraform plan in their local dev environment to see if anything surprising happens. At this point it's fine to abort and return to the original version without committing anything.
• That person commits the change to required_version and submits a pull request (or equivalent) to the rest of the team, which gives the team an opportunity to coordinate accepting that change.
• After the pull request is accepted and merged, everyone on the team upgrades Terraform and the state will be updated after the next terraform apply.

Some teams handle this in a more formal way by running Terraform in a central automation system where that system controls which version of Terraform is used, which is one way to make this a technology solution rather than a social (via version control) solution, but that's not required.

If more testing is desired, Terraform's workspaces feature is intended as a way to create a temporary "copy" of your infrastructure that can be used to test changes, at the expense of some extra steps. Workspaces are usually correlated roughly with version control branches, so the above process would begin with the extra steps of creating a new branch and new workspace, applying the branch configuration on the old version of Terraform, and then doing the above steps in the branch. At the end, there are extra steps of destroying the temporary workspace along with its associated branch and then merging the change down to the main branch before using it "for real". In practice this is often overkill for a Terraform Core upgrade, but is an option.

The above approaches can also be used with the provider plugin version constraint mechanisms in order to coordinate provider upgrades.

---

On the subject of state backups: Terraform currently delegates that problem for each backend to solve, though indeed some of them do this better than others.

• If you are using the "local" backend (not recommended for a team situation) then it will write out terraform.tfstate.backup for each change, giving you one level of "undo" on the state.
• For the Amazon S3 backend, users will generally enable versioning on the target bucket, allowing any historical state version to be recovered.
• The Terraform Enterprise backend (for historical reasons, named "atlas") captures all historical versions by default, as a standard feature.

At this time some backends do not have a facility for historical backups at all. In the short term, it's unfortunately up to users to select one that does if that is important to them. Over time, I expect that we will ensure that there's a plausible backup story for all backends and consider gradually phasing out those few where such a thing would be technically impossible.

Rolling back to an older state is generally a last resort, since it will cause Terraform to lose track of any objects that were created since that state snapshot was created, but it is certainly useful to do this in some cases.

---

I hope that helps. If I've missed the point of what you're trying to achieve here, please do let me know!

[hdost]

So we are already using a pipeline for application of the terraform templates. Additionally we have not seen issues since we pinned so it may be a moot point. As background we are using azure as the backend.

It may be too heavy handed to have both an argument as well as version pinning.

[tcarrio]

@hdost I'm curious whether the argument could be used for version pinning?

The behavior for the --update-state=false flag being that the current version of Terraform will be pinned to the configuration.

Effectively a short-handed manner of applying the suggested workflow from the CLI.

[hdost]

I like that idea.

@apparentlymart does that make sense given what's already been discussed?

[hdost]

Alright so after working with terraform a bit more. I have what think might be more pragmatic would be to do would be add a warning if there is not a version specified for the required_version. To encourage better practices.

@apparentlymart @mildwonkey What are your thoughts?