Terraform doesn't apply changes to column length in bigquery table despite having been excecuted succesfully

esteban-castillo commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

terraform version
Terraform v1.2.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v4.28.0

Affected Resource(s)

google_bigquery_table

Terraform Configuration Files

Supose we have a single bigquery table deployed with the following configuration

resource google_bigquery_table "table" {
  table_id = "my_table"
  dataset_id = "my_dataset"
  schema = file("schema.json")
}

With the following schema.json defining a column with a certain length

[
    {
        "maxLength": "50",
        "mode": "REQUIRED",
        "name": "MY_COLUMN",
        "type": "STRING"
    }
]

Then, we generate a change to a different maxLength, for instance:

[
    {
        "maxLength": "10",
        "mode": "REQUIRED",
        "name": "MY_COLUMN",
        "type": "STRING"
    }
]

Debug Output

Terraform detects the changes in the plan step

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_bigquery_table.table will be updated in-place
  ~ resource "google_bigquery_table" "table" {
        id                  = "projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table"
      ~ schema              = jsonencode(
          ~ [
              ~ {
                  ~ maxLength = "50" -> "10"
                    name      = "NROCONTRATO"
                    # (2 unchanged elements hidden)
                },
            ]
        )
        # (15 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

and reports succesful changes in apply

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_bigquery_table.table: Modifying... [id=projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table]
google_bigquery_table.table: Modifications complete after 1s [id=projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Expected Behavior

The deployed table should be modified to have maxLenght=10

Actual Behavior

The deployed table is not changed and the changes arent reflected on the terraform state either

Steps to Reproduce

Deploy configuration with the first schema file
Deploy configuration with the modified schema

b/300616520

edwardmedia commented 2 years ago

@esteban-castillo can you share your debug log that contains the api requests & responses?

esteban-castillo commented 2 years ago

022-07-14T18:13:50.076Z [INFO]  backend/local: apply calling Apply
2022-07-14T18:13:50.076Z [DEBUG] Building and walking apply graph for NormalMode plan
2022-07-14T18:13:50.076Z [DEBUG] adding implicit provider configuration provider["registry.terraform.io/hashicorp/google"], implied first by google_bigquery_table.table (expand)
2022-07-14T18:13:50.076Z [DEBUG] ProviderTransformer: "google_bigquery_table.table" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/hashicorp/google"]
2022-07-14T18:13:50.076Z [DEBUG] ProviderTransformer: "google_bigquery_table.table (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"]
2022-07-14T18:13:50.076Z [DEBUG] ReferenceTransformer: "google_bigquery_table.table (expand)" references: []
2022-07-14T18:13:50.076Z [DEBUG] ReferenceTransformer: "google_bigquery_table.table" references: []
2022-07-14T18:13:50.076Z [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: []
2022-07-14T18:13:50.076Z [DEBUG] Starting graph walk: walkApply
2022-07-14T18:13:50.076Z [DEBUG] created provider logger: level=debug
2022-07-14T18:13:50.077Z [INFO]  provider: configuring client automatic mTLS
2022-07-14T18:13:50.103Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.28.0/linux_amd64/terraform-provider-google_v4.28.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/google/4.28.0/linux_amd64/terraform-provider-google_v4.28.0_x5]
2022-07-14T18:13:50.104Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.28.0/linux_amd64/terraform-provider-google_v4.28.0_x5 pid=5772
2022-07-14T18:13:50.105Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.28.0/linux_amd64/terraform-provider-google_v4.28.0_x5
2022-07-14T18:13:50.117Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: configuring server automatic mTLS: timestamp=2022-07-14T18:13:50.117Z
2022-07-14T18:13:50.187Z [DEBUG] provider.terraform-provider-google_v4.28.0_x5: plugin address: address=/tmp/plugin054625327 network=unix timestamp=2022-07-14T18:13:50.187Z
2022-07-14T18:13:50.189Z [DEBUG] provider: using plugin: version=5
2022-07-14T18:13:50.483Z [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/hashicorp/google\"]" changed the config value, but that value is unused
2022-07-14T18:13:50.487Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO] Authenticating using DefaultClient...: timestamp=2022-07-14T18:13:50.487Z
2022-07-14T18:13:50.488Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: timestamp=2022-07-14T18:13:50.488Z
2022-07-14T18:13:50.489Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO] Authenticating using DefaultClient...: timestamp=2022-07-14T18:13:50.489Z
2022-07-14T18:13:50.489Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: timestamp=2022-07-14T18:13:50.489Z
2022-07-14T18:13:50.489Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Waiting for state to become: [success]: timestamp=2022-07-14T18:13:50.489Z
2022-07-14T18:13:50.581Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO] Terraform is using this identity: wom-edms-dev@wom-edms-dev-064f.iam.gserviceaccount.com: timestamp=2022-07-14T18:13:50.581Z
2022-07-14T18:13:50.593Z [WARN]  Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for google_bigquery_table.table, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .deletion_protection: planned value cty.True for a non-computed attribute
      - .description: planned value cty.StringVal("") for a non-computed attribute
      - .schema: planned value cty.StringVal("[{\"maxLength\":\"10\",\"mode\":\"REQUIRED\",\"name\":\"NROCONTRATO\",\"type\":\"STRING\"}]") does not match config value cty.StringVal("[\n    {\n        \"maxLength\": \"10\",\n        \"mode\": \"REQUIRED\",\n        \"name\": \"NROCONTRATO\",\n        \"type\": \"STRING\"\n    }\n]") nor prior value cty.StringVal("[{\"maxLength\":\"50\",\"mode\":\"REQUIRED\",\"name\":\"NROCONTRATO\",\"type\":\"STRING\"}]")
      - .friendly_name: planned value cty.StringVal("") for a non-computed attribute
      - .labels: planned value cty.MapValEmpty(cty.String) for a non-computed attribute
google_bigquery_table.table: Modifying... [id=projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table]
2022-07-14T18:13:50.604Z [INFO]  Starting apply for google_bigquery_table.table
2022-07-14T18:13:50.605Z [DEBUG] google_bigquery_table.table: applying the planned Update change
2022-07-14T18:13:50.612Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO] Updating BigQuery table: projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table: timestamp=2022-07-14T18:13:50.609Z
2022-07-14T18:13:50.616Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO] Instantiating Google Cloud BigQuery client for path https://bigquery.googleapis.com/bigquery/v2/: timestamp=2022-07-14T18:13:50.609Z
2022-07-14T18:13:50.619Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2022-07-14T18:13:50.610Z
2022-07-14T18:13:50.621Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: request attempt 0: timestamp=2022-07-14T18:13:50.610Z
2022-07-14T18:13:50.621Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2022-07-14T18:13:50.612Z
2022-07-14T18:13:50.622Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: request attempt 0: timestamp=2022-07-14T18:13:50.612Z
2022-07-14T18:13:50.622Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
PUT /bigquery/v2/projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table?alt=json&prettyPrint=false HTTP/1.1
Host: bigquery.googleapis.com
User-Agent: google-api-go-client/0.5 Terraform/1.2.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/dev
Content-Length: 193
Content-Type: application/json
X-Goog-Api-Client: gl-go/1.16.14 gdcl/0.82.0
Accept-Encoding: gzip
{
 "schema": {
  "fields": [
   {
    "maxLength": "10",
    "mode": "REQUIRED",
    "name": "NROCONTRATO",
    "type": "STRING"
   }
  ]
 },
 "tableReference": {
  "datasetId": "DATA_OPS",
  "projectId": "wom-edms-dev-064f",
  "tableId": "my_table"
 }
}
-----------------------------------------------------: timestamp=2022-07-14T18:13:50.615Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Thu, 14 Jul 2022 18:13:50 GMT
Etag: b19PGruHKIlS4zsHn0QV4Q==
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0
{
 "kind": "bigquery#table",
 "etag": "b19PGruHKIlS4zsHn0QV4Q==",
 "id": "wom-edms-dev-064f:DATA_OPS.my_table",
 "selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table",
 "tableReference": {
  "projectId": "wom-edms-dev-064f",
  "datasetId": "DATA_OPS",
  "tableId": "my_table"
 },
 "schema": {
  "fields": [
   {
    "name": "NROCONTRATO",
    "type": "STRING",
    "mode": "REQUIRED",
    "maxLength": "50"
   }
  ]
 },
 "numBytes": "0",
 "numLongTermBytes": "0",
 "numRows": "0",
 "creationTime": "1657818022016",
 "lastModifiedTime": "1657818022051",
 "type": "TABLE",
 "location": "US",
 "numTotalLogicalBytes": "0",
 "numActiveLogicalBytes": "0",
 "numLongTermLogicalBytes": "0"
}
-----------------------------------------------------: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO] Reading BigQuery table: projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [INFO] Instantiating Google Cloud BigQuery client for path https://bigquery.googleapis.com/bigquery/v2/: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: request attempt 0: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: request attempt 0: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.753Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /bigquery/v2/projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table?alt=json&prettyPrint=false HTTP/1.1
Host: bigquery.googleapis.com
User-Agent: google-api-go-client/0.5 Terraform/1.2.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/dev
X-Goog-Api-Client: gl-go/1.16.14 gdcl/0.82.0
Accept-Encoding: gzip
-----------------------------------------------------: timestamp=2022-07-14T18:13:50.752Z
2022-07-14T18:13:50.833Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Thu, 14 Jul 2022 18:13:50 GMT
Etag: b19PGruHKIlS4zsHn0QV4Q==
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0
{
 "kind": "bigquery#table",
 "etag": "b19PGruHKIlS4zsHn0QV4Q==",
 "id": "wom-edms-dev-064f:DATA_OPS.my_table",
 "selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table",
 "tableReference": {
  "projectId": "wom-edms-dev-064f",
  "datasetId": "DATA_OPS",
  "tableId": "my_table"
 },
 "schema": {
  "fields": [
   {
    "name": "NROCONTRATO",
    "type": "STRING",
    "mode": "REQUIRED",
    "maxLength": "50"
   }
  ]
 },
 "numBytes": "0",
 "numLongTermBytes": "0",
 "numRows": "0",
 "creationTime": "1657818022016",
 "lastModifiedTime": "1657818022051",
 "type": "TABLE",
 "location": "US",
 "numTotalLogicalBytes": "0",
 "numActiveLogicalBytes": "0",
 "numLongTermLogicalBytes": "0"
}
-----------------------------------------------------: timestamp=2022-07-14T18:13:50.832Z
2022-07-14T18:13:50.834Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2022-07-14T18:13:50.832Z
2022-07-14T18:13:50.834Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-07-14T18:13:50.832Z
2022-07-14T18:13:50.834Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2022-07-14T18:13:50.832Z
2022-07-14T18:13:50.834Z [INFO]  provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-07-14T18:13:50.832Z
2022-07-14T18:13:50.834Z [WARN]  Provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value for google_bigquery_table.table, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .schema: was cty.StringVal("[{\"maxLength\":\"10\",\"mode\":\"REQUIRED\",\"name\":\"NROCONTRATO\",\"type\":\"STRING\"}]"), but now cty.StringVal("[{\"maxLength\":\"50\",\"mode\":\"REQUIRED\",\"name\":\"NROCONTRATO\",\"type\":\"STRING\"}]")
google_bigquery_table.table: Modifications complete after 0s [id=projects/wom-edms-dev-064f/datasets/DATA_OPS/tables/my_table]
2022-07-14T18:13:50.866Z [DEBUG] provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] [core] [Server #1 ListenSocket #2] ListenSocket deleted 
2022-07-14T18:13:50.867Z [DEBUG] provider.terraform-provider-google_v4.28.0_x5: 2022/07/14 18:13:50 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /tmp/plugin054625327->@: use of closed network connection 
2022-07-14T18:13:50.867Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.28.0/linux_amd64/terraform-provider-google_v4.28.0_x5 pid=5772
2022-07-14T18:13:50.867Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2022-07-14T18:13:50.867Z [DEBUG] provider: plugin exited

edwardmedia commented 2 years ago

b/239072731

edwardmedia commented 2 years ago

@esteban-castillo update maxLength from 50 to 10 is not allowed. Can you try from 10 to 50 instead?

https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#alter_column_set_data_type_statement

esteban-castillo commented 2 years ago

Yes, we tried that as well and the behaviour is the same

hashicorp / terraform-provider-google