Improve versioning and tracking of tfstate changes in Git commit and merge

[juarezr]

Improve versioning and tracking of tfstate changes in Git commit and merge

Current Terraform Version

terraform {
  required_version = ">=1.0.11"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=2.85.0"
    }
}

Use-cases

Currently, we have many terraform repositories that are separated by the specific needs of the application or the area of impact on our cloud infrastructure.

For tracking the state we use two strategies/use cases:

1. terraform_remote_state for repositories/areas that are larger, complex, or have big impact.
2. Local terraform.tstate files tracked in git for smaller, simpler apps for reducing the burden of provisioning, backup, and maintenance.

Problem description

Currently, when using local state files on git, like (2) above, we have a big changeset between two terraform apply that spans the whole lines of the terraform.tfstate file even when the real change is just one property.

This complicates tracking changes and merging in the use case (2).

The problem is that between two terraform apply commands:

1. the dependencies change their position inside the terraform.tfstate in a random order.
2. the id of data blocks/attributes change according to a timestamp even if their properties are unchanged.

See:

@@ -159,10 +223,10 @@
           "sensitive_attributes": [],
           "dependencies": [
+            "data.azurerm_virtual_network.vnet_root",
             "azurerm_resource_group.rg_proj",
             "azurerm_storage_account.storage_account_proj",
-            "data.azurerm_client_config.current",
-            "data.azurerm_virtual_network.vnet_root"
+            "data.azurerm_client_config.current"
           ]

And:

@@ -15,7 +24,7 @@
   "resources": [
     {
         "mode": "data",
           "schema_version": 0,
           "attributes": {
             "client_id": "04b07795-8ddb-461a-bbee-02f9e1bf7b46",
-            "id": "2021-11-22 15:02:12.055416 +0000 UTC",
+            "id": "2021-12-02 21:46:39.616746 +0000 UTC",
             "object_id": "11111111-aaaa-2222-bbbb-333333333333",
             "subscription_id": "44444444-cccc-6666-dddd-555555555555",
             "tenant_id": "77777777-eeee-8888-ffff-999999999999",

Attempted Solutions

For dealing with this problem we had been:

1. Reverting the lines unchanged lines of dependency and `id´
2. Committing the changes and handling the conflits further when merging

Proposal

For reducing the changeset I would like to propose changes that IMHO don't impact on compatibility:

Sorting dependency in alphabetical order before writing to terraform.tfstate

This will improve because:

1. Unchanged dependencies will not cause line changes in the Git diff algorithm.
2. New dependencies will be placed in a way that won't cause line changes in further commits/changesets.

See:

           "sensitive_attributes": [],
           "dependencies": [
             "azurerm_resource_group.rg_proj",
             "azurerm_storage_account.storage_account_proj",
!            "data.azurerm_client_config.current",
!            "data.azurerm_virtual_network.vnet_root"
           ]

Avoid changing the id of data blocks when nothing else has changed

The data didn't change since the last timestamp anyway:

   "resources": [
     {
       "mode": "data",
           "schema_version": 0,
           "attributes": {
             "client_id": "04b07795-8ddb-461a-bbee-02f9e1bf7b46",
!            "id": "2021-11-22 15:02:12.055416 +0000 UTC",
             "object_id": "11111111-aaaa-2222-bbbb-333333333333",
             "subscription_id": "44444444-cccc-6666-dddd-555555555555",
             "tenant_id": "77777777-eeee-8888-ffff-999999999999",

[jbardin]

Thanks for the request @juarezr,

The order of the dependencies is not taken into account within Terraform, so it seems reasonable that we could sort these as part of the serialization process to help with data normalization.

The id fields however are part of the data source, and not handled independently by Terraform (the fact that everything has an id is only due to the provider SDK). The problematic data sources would need to be individually updated by each provider to prevent any changes from being recorded. There are also numerous other resources that update fields each time they are read which will cause the same type of change. While these may cause a nuisance generating noise in the state and "Objects have changed outside of Terraform", it is not incorrect for providers to do this, so it always will be something that must be dealt with.

If you are concerned with only tracking changes made by the configuration, and you are running Terraform from a centralized workflow, it perhaps may be better to track the output of the plans directly which will show planned changes separately from external drift.

[apparentlymart]

In addition to what @jbardin said, while there would certainly not be any harm in making the parts of this that Terraform Core controls serialize in a consistent way, I want to be clear that keeping your Terraform state under version control is not a recommended workflow and so Terraform is not designed around that usage pattern. The only currently-supported approach for sharing state across a team is to use a remote state storage backend.

I wouldn't expect that recommendation to change even if we do make the state snapshot serialization use a normalized ordering, because it still doesn't address the fundamental workflow problem that you can't know the new state until you've applied the changes, but you typically shouldn't apply the changes until they are pushed to your version control system or else others might make conflicting changes that don't take yours into account and thus create a difficult-to-reconcile fork. There's some older discussion about that in #13891.

[juarezr]

I completely agree that storing state in version control neither scales nor solves any team workflow need.

Also, it could be even dangerous besides isolated and temporary scenarios like proof-of-concept or experimentation by a single person

The only way it couldn't be worse it's leaving/losing the tfstate in your local computer.