google_bigquery_table - view sql update clears descriptions of all columns

[dequire]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.6.4 on darwin_arm64

• provider registry.terraform.io/hashicorp/google v5.7.0

Affected Resource(s)

google_bigquery_table

Terraform Configuration Files

resource "google_bigquery_table" "dataset_views" {
  for_each = local.dataset_views

  project             = var.datahub_project
  dataset_id          = google_bigquery_dataset.dataset.dataset_id
  table_id            = split(".", lower(each.value))[0]
  deletion_protection = false

  # view definition
  view {
    query          = templatefile("${local.dataset_definition_path}/${each.value}", local.template_vars)
    use_legacy_sql = false
  }
  schema = try(templatefile("${local.dataset_definition_path}/${split(".", each.value)[0]}.schema", local.template_vars), null)
  depends_on = [google_bigquery_dataset.dataset]
}

Debug Output

Too long - attached as file. terraform-debug-log.txt

Expected Behavior

A new column should be added to the view named "DUMMY" with description "Test Marek"

Actual Behavior

New column is added, but description is blank - for all columns, not just the one which was added.

Steps to Reproduce

1. Add new column to SQL and schema
2. Run terraform apply. The plan displayed is correct with expected change to both SQL and schema
3. New column is added, but descriptions are not.

Important Factoids

There is one suspicious line in log, which ways that invalid plan was produced.

2023-12-01T13:00:58.903+0100 [WARN]  Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for module.semn_pos_offline.google_bigquery_table.dataset_views["TEST_TABLE.sql"], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .labels: planned value cty.MapValEmpty(cty.String) for a non-computed attribute
      - .friendly_name: planned value cty.StringVal("") for a non-computed attribute
      - .max_staleness: planned value cty.StringVal("") for a non-computed attribute
      - .description: planned value cty.StringVal("") for a non-computed attribute
      - .require_partition_filter: planned value cty.False for a non-computed attribute
      - .schema: planned value cty.StringVal("[{\"description\":\"Date of the Day\",\"mode\":\"NULLABLE\",\"name\":\"DATE_KEY\",\"type\":\"DATE\"},{\"description\":\"Product key\",\"mode\":\"NULLABLE\",\"name\":\"ITEM_KEY\",\"type\":\"INTEGER\"},{\"description\":\"Test Marek\",\"mode\":\"NULLABLE\",\"name\":\"DUMMY\",\"type\":\"INTEGER\"}]") does not match config value cty.StringVal("[\n    {\n        \"name\": \"DATE_KEY\",\n        \"type\": \"DATE\",\n        \"mode\": \"NULLABLE\",\n        \"description\": \"Date of the Day\"\n    },\n    {\n        \"name\": \"ITEM_KEY\",\n        \"type\": \"INTEGER\",\n        \"mode\": \"NULLABLE\",\n        \"description\": \"Product key\"\n    },\n    {\n        \"name\": \"DUMMY\",\n        \"type\": \"INTEGER\",\n        \"mode\": \"NULLABLE\",\n        \"description\": \"Test Marek\"\n    }\n]\n") nor prior value cty.StringVal("[{\"description\":\"Date of the Day\",\"mode\":\"NULLABLE\",\"name\":\"DATE_KEY\",\"type\":\"DATE\"},{\"description\":\"Product key\",\"mode\":\"NULLABLE\",\"name\":\"ITEM_KEY\",\"type\":\"INTEGER\"}]")

References

b/314616737

[edwardmedia]

It appears the api drops the description.

schema in the request

"schema": {
 "fields": [
  {
   "description": "Date of the Day",
   "mode": "NULLABLE",
   "name": "DATE_KEY",
   "type": "DATE"
  },
  {
   "description": "Product key",
   "mode": "NULLABLE",
   "name": "ITEM_KEY",
   "type": "INTEGER"
  },
  {
   "description": "Test Marek",
   "mode": "NULLABLE",
   "name": "DUMMY",
   "type": "INTEGER"
  }
 ]
},

schema in the response

"schema": {
 "fields": [
  {
   "name": "DATE_KEY",
   "type": "DATE",
   "mode": "NULLABLE"
  },
  {
   "name": "ITEM_KEY",
   "type": "INTEGER",
   "mode": "NULLABLE"
  },
  {
   "name": "DUMMY",
   "type": "INTEGER",
   "mode": "NULLABLE"
  }
 ]
},

[takotakot]

I think this is similar to #12616 .

When I try to terraform apply again, the descriptions are updated.

[dequire]

@takotakot , sounds about right. We're also doing double apply to work around this issue currently.