search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
41.76k stars 9.42k forks source link

Terraform lies about the terraform_version in `terraform state pull` (s3 backend) #25866

Open Moeser opened 3 years ago

Moeser commented 3 years ago

Terraform Version

0.12.29

What I run

$ ~/bin/terraform-0.12.29 state pull | grep terraform_version # what tf 0.12.29 thinks the version is
  "terraform_version": "0.12.29",
$ ~/bin/terraform-0.12.28 state pull | grep terraform_version # what tf 0.12.28 thinks the version is
  "terraform_version": "0.12.28",
$ aws s3 cp s3://<mybucket>/<myprefix>/terraform.tfstate ./local.tfstate
download: s3://<mybucket>/<myprefix>/terraform.tfstate to ./local.tfstate
$ grep terraform_version local.tfstate                        # what the actual version is
  "terraform_version": "0.12.26",

Expected Behavior

I expect Terraform to tell me the actual version in the remote state and not lie to me. Why is it processing the json? Shouldn't it just dump the json as-is from s3?

Actual Behavior

Terraform lied to me. It edited the state to show whatever version of binary I was running instead of the real version in the state living in s3. My feelings were hurt by Terraform's lies. I lost trust in my once trustworthy sidekick.

Steps to Reproduce

  1. Store a remote state in s3 with an OLDER version of Terraform 1.a. You have to use an OLDER version because Terraform refuses to show a newer state in terraform state pull, which is a totally different but related issue. This seems silly since Terraform could probably just output the json instead of processing it.
  2. Run state pull from a NEWER version of Terraform: terraform state pull | grep terraform_version
  3. Witness Terraform's nose grow longer as it lies like Pinocchio

Additional Context

I haven't tried this for any other backends, so it may be specific to the s3 backend.

In case it's helpful: here's the output if we try to view a state with a NEWER version of Terraform than the currently running binary:

$ ~/bin/terraform-0.12.24 state pull | grep terraform_version
Failed to refresh state: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state
$ echo "duuuude, i'm just trying to see the json" | ~/bin/terraform-0.12.24 state pull
Failed to refresh state: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state
$ echo "failed to refresh state? is this another one of your lies?" | ~/bin/terraform-0.12.24 state pull
Failed to refresh state: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state
$ echo "but you didn't fail, you saw the version.  can you just print the rest?" | ~/bin/terraform-0.12.24 state pull
Failed to refresh state: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state
$ echo "you forget how to output json if you see a new version string?" | ~/bin/terraform-0.12.24 state pull
Failed to refresh state: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state
$ echo "are you refusing because you're jealous of newer versions?" | ~/bin/terraform-0.12.24 state pull
Failed to refresh state: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state
$ echo "why you so jelly?" | ~/bin/terraform-0.12.24 state pull
Failed to refresh state: state snapshot was created by Terraform v0.12.26, which is newer than current v0.12.24; upgrade to Terraform v0.12.26 or greater to work with this state

You can try to reason with it, but the conversation ends up pretty one sided.

I feel like these are two symptoms of the same problem. Terraform is processing the state during terraform state pull and should probably just output it directly instead of reading or manipulating the content, or lying to me like a dirty lying liar.

Should it be reading or manipulating the content? Well, lets ask Terraform:

$ terraform-0.12.29 state pull --help
Usage: terraform state pull [options]

  Pull the state from its location and output it to stdout.

  This command "pulls" the current state and outputs it to stdout.
  The primary use of this is for state stored remotely. This command
  will still work with local state but is less useful for this.

Weirdly, it doesn't say "sneakily edits the json to change the version" or "peeks at the version string and forgets how to simply output json when seeing a newer one". How strange! :)

References

jbardin commented 3 years ago

Thanks for filing the issue @Moeser

Yes, Terraform currently only has one method for reading the state which entails decoding it. In order to decode the state, it must go through any state upgrade paths which sets the state and terraform versions. The remote state API only provides access to the decoded state, so direct file access isn't really possible at this time.

Since this is working as intended, and the desired functionality is going to take some design work, I'm going to re-tag this as an enhancement for now.

sunghospark-calm commented 3 years ago

@jbardin this feels like more of a bug then enhancement to me. At the very least I think this should be documented in cli and website.

The terraform state pull command is used to manually download and output the state from remote state. This command also works with local state.

No where in the description it says about changing the state file. This can be really hectic during the terraform upgrade process especially when encountered other issues such as https://github.com/hashicorp/terraform/issues/23290

geekofalltrades commented 3 years ago

This gave me a minor heart attack while preparing to upgrade from 0.12 to 0.13. I ran init and plan with 0.13 on 0.13-upgraded code to check for any problems, then ran terraform state pull to sanity-check that I hadn't just tainted the state file for all of my colleagues and automated processes still running 0.12.21, and Terraform appeared to tell me that I had done exactly that. I ran to the S3 state bucket prepared to roll back to the last version of the state file only to find that it hadn't actually changed.

I had previously tested on a local state file and relied on git diff, which showed me that the state file hadn't changed after a plan with 0.13, which is what I was expecting, so I was extra confused.

jcollum commented 3 years ago

In order to decode the state, it must go through any state upgrade paths which sets the state and terraform versions

How do I find the "state upgrade paths" that are causing TF to think that the state was created with a newer version? This issue has me stuck in a loop.