Changing only the `settings.version` attribute in the `google_sql_database_instance` resource should not be shown as drift

verdel commented 2 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

Terraform v1.7.5 on darwin_arm64

provider registry.terraform.io/hashicorp/google v5.24.0
provider registry.terraform.io/hashicorp/google-beta v5.24.0

Affected Resource(s)

google_sql_database_instance

Terraform Configuration

resource "google_sql_database_instance" "default" {...}
output "settings" {
  value       = google_sql_database_instance.default.settings
}

Debug Output

No response

Expected Behavior

Changing the settings.version attribute should not cause drift in the google_sql_database_instance resource if other attributes in settings are not changed

Actual Behavior

Several users have reported cases where only the settings.version attribute was changed, without altering any other settings attributes. Since the settings.version attribute is a read-only attribute, its external modification should not be reflected as drift in the configuration of the google_sql_database_instance resource.

Here's what the output of the terraform plan command looks like in this case:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:

    create

Terraform will perform the following actions:
module.pg.google_sql_database.default[0] will be created

    resource "google_sql_database" "default" {
        charset = "UTF8"
        collation = "en_US.UTF8"
        deletion_policy = "DELETE"
        id = (known after apply)
        instance = "******-stage-db-02df5265"
        name = "******-stage-db"
        project = "******-stage"
        self_link = (known after apply)
        }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

module.pg.google_sql_database_instance.default has changed

~ resource "google_sql_database_instance" "default" { 
     id = "******-stage-db-02df5265"
     name = "******-stage-db-02df5265"
     # (14 unchanged attributes hidden)

  ~ settings {
       ~ version = 18 -> 21
          # (12 unchanged attributes hidden)

          # (5 unchanged blocks hidden)
     }

    # (1 unchanged block hidden)
}

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes

Steps to reproduce

terraform apply or terraform plan

Important Factoids

No response

References

https://github.com/terraform-google-modules/terraform-google-sql-db/issues/548

b/335253065

ggtisc commented 2 months ago

Hi @verdel!

As I understand reading the main conversation between you and imrannayer from the attached link and from this one:

You have an existing google_sql_database resource, but then when you update any other resources that aren't the google_sql_database it internally changes the settings.version

Please confirm this statement to understand your needs.

verdel commented 2 months ago

@ggtisc, not quite. I'm creating a google_sql_database_instance(not google_sql_database) resource, and after some time when executing terraform plan or terraform apply, I get a warning in the command output indicating that the settings.version attribute of the resource has changed. Meanwhile, no other attributes within settings change (although, based on the purpose of settings.version— which is to be an incrementing revision number of settings — this attribute should not change unless another settings key changes).

The warning that I receive in the command output:

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan

ggtisc commented 2 months ago

Then you have the existing google_sql_database_instanceresource without making changes for a long time. And one day without making any change again to that same resource or adding other new resources to that project you simply run aterraform apply or a terraform plan and it change the version of this unique resource? or were there changes to other resources in the same project?

verdel commented 2 months ago

@ggtisc, the Terraform code does not change between calls to terraform apply. No changes are made in the web interface or through gcloud either. Nothing changes in other resources within the project. The output of terraform plan or terraform apply indicates that only the settings.version attribute changes.

In the JSON API documentation, it is written that settingsVersion is the version of instance settings. This means that changes to this attribute(settings.version) can only occur if the settings object itself changes.

I thought that one possible reason for the change in settings.version might be that some sub-attributes present in a Google Cloud PostgreSQL instance are not being saved in the Terraform state. If this sub-attribute changes on the platform side, then settings.version will change, but since the sub-attribute is beyond the visibility for the Terraform state, in the output of terraform plan or terraform apply we will only see the change in settings.version.

I checked in the resource_sql_database_instance.go code all the sub-attributes that are saved in the Terraform state and noticed a discrepancy with the documentation only in one attribute - databaseReplicationEnabled. This is just my guess, but it is the only difference from the documentation that I see right now.

Unfortunately, I can't yet explain why the real state of a Google Cloud PostgreSQL instance might only differ in settings.version from the Terraform state.

I am currently waiting for the settings.version change to be reproduced without changes in other sub-attributes.

As I currently do not have my own data to confirm this behavior, in the process of creating this issue, I requested data from another participant(@juliusoh) in the discussion of the issue in the terraform-google-modules/terraform-google-sql-db repository.

ggtisc commented 2 months ago

It looks like the 'settrings.version' is being updated randomly without changing any configuration in the involved resources on a main-configuration.tf but isn't possible to replicate this scenario to be sure that it is happening. As user reported any change was made to the resources, just made a terraform plan or terraform apply without changes to be applied.

verdel commented 2 months ago

Let's wait a bit. As soon as I reproduce the issue, I will add additional information, including the output of the command gcloud sql instances describe <db-instance-name> before and after changing settings.version.

By comparing the outputs, we can see exactly what changes in the settings.

verdel commented 2 months ago

@ggtisc, I finally received confirmation that the settings.version can change without altering other attributes of the Google Cloud SQL instance.

Console output of terraform apply:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # google_sql_database_instance.default has changed
  ~ resource "google_sql_database_instance" "default" {
        id                             = "test"
        name                           = "test"
        # (18 unchanged attributes hidden)

      ~ settings {
          ~ version                     = 99 -> 101
            # (14 unchanged attributes hidden)

            # (6 unchanged blocks hidden)
        }

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Changes to Outputs:
  ~ primary                  = (sensitive value)

Diff between outputs of gcloud sql instances describe --format json test on version 99 and 101:

10c10
<   "etag": "c95ddf6f008ad754520321e587f5fa4d763a0e744c9494c96d70347c9fe79837",
---
>   "etag": "c7862a289e42ba50bed4e68c11bc0f6f1cef281b1250f3449335b1d3c6373747",
95c95
<     "settingsVersion": "99",
---
>     "settingsVersion": "101",

The diff shows that only the settingsVersion changes, which in terraform provider is converted into the settings.version attribute, and etag, an unused attribute in the terraform provider that is deprecated and was previously used instead of settingsVersion.

If necessary, I can provide the debug output of terraform apply and the full output of gcloud sql instances describe --format json.

dullest commented 3 weeks ago

I have faced exactly same diff like following several times. I also have not changed the terraform codes for it.

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.postgresql.google_sql_database_instance.default has changed
! resource "google_sql_database_instance" "default" {
        id                             = "foo-252027d3"
        name                           = "foo-252027d3"
        # (14 unchanged attributes hidden)

!       settings {
!           version                     = 46 -> 48
            # (12 unchanged attributes hidden)

            # (4 unchanged blocks hidden)
        }

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

hashicorp / terraform-provider-google