google_bigquery_dataset - storage_billing_model is always shown as "changed"

[AndreaGiardini]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.2.5 on darwin_arm64

• provider registry.terraform.io/hashicorp/google v4.75.0

Affected Resource(s)

• google_bigquery_dataset

Terraform Configuration Files

resource "google_bigquery_dataset" "this" {
  dataset_id  = var.dataset_id
  description = var.description
  location    = coalesce(var.non_standard_location, "EU")

  default_partition_expiration_ms = var.default_partition_expiration_ms

  access {
    role          = "OWNER"
    special_group = "projectOwners"
  }

  lifecycle {
    ignore_changes = [access]
  }

  delete_contents_on_destroy = var.dataset_delete_contents_on_destroy
  storage_billing_model      = var.dataset_storage_billing_model
}

Expected Behavior

After setting var.dataset_storage_billing_model to LOGICAL (the default). The change should be applied and following runs should not show any changes.

Actual Behavior

Runs always show an in-place change, even if the terraform plan is applied correctly.

  # module.dwh_sources.module.bigquery_datasets["ABC_DATA"].google_bigquery_dataset.this will be updated in-place
  ~ resource "google_bigquery_dataset" "this" {
        id                              = "projects/abc-dwh-dev/datasets/ABC_DATA"
      + storage_billing_model           = "LOGICAL"

[edwardmedia]

b/294082700

[LucaPrete]

Hello @AndreaGiardini.

I was able to reproduce this and we opened an internal bug. I'll try to figure out what's going on. Either me or other Googlers will send a patch for this asap.

In the meantime, I'd recommend to use -once the argument has been set-

lifecycle {
  ignore_changes = [
    storage_billing_model
  ]
}

to workaround the issue.

[LucaPrete]

Also @AndreaGiardini, LOGICAL should be the default if the value is not set. So you might opt to simply not specifying it to avoid any drift.

[AndreaGiardini]

Not sure if that's relevant to the issue, but it's still something that confused me while investigating it: why are we using a String rather than an Enum here? https://github.com/GoogleCloudPlatform/magic-modules/pull/7615/files#diff-477c9626862167a139bdfed6809661472c7cc60cdf76dc5ad98056e2e17e4ca3R359-R367

The Google API specification for storageBillingModel is Enum.

[AndreaGiardini]

Hi @LucaPrete

The workaround you specified is a bit cumbersome to apply on our side since we use this resource in a module and it's used to create many different datasets (some of them with storage_billing_model=LOGICAL and some of them PHYSICAL). Ignoring the change or setting it to null has all kinds of repercussions on our side.

Do you have an estimation for a fix? Is there any way I can help move it forward?

[LucaPrete]

Hello @AndreaGiardini unfortunately this is not something I can fix in Terraform but it's how the API currently works. TL;DR if you set logical -even straight from APIs- this is not returned in the following GET. We're investigating with the product team to understand how to evolve this.

What if you just put null as value in your module? This should still set logical while not causing the drift...

[edwardmedia]

@AndreaGiardini the api has deployed a fix in the backend. Can you verify and confirm the issue is fixed?

[jokester]

Not happening for me today with provider version = "5.11.0"