Terraform Provider not creating BigQuery View Schema correctly, all fields have mode:NULLABLE

ds8080 commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

We are using terraform-enterprise to deploy the terraform to our GCP projects Terraform Provider Version : 4.59.0 Terraform version : 1.2.3

Affected Resource(s)

google_bigquery_table

Terraform Configuration Files

# Copy-paste your Terraform configurations here.
#
# For large Terraform configs, please use a service like Dropbox and share a link to the ZIP file.
# For security, you can also encrypt the files using our GPG public key:
#    https://www.hashicorp.com/security
#
# If reproducing the bug involves modifying the config file (e.g., apply a config,
# change a value, apply the config again, see the bug), then please include both:
# * the version of the config before the change, and
# * the version of the config after the change.

resource "google_bigquery_dataset" "my_dataset" {
  dataset_id = "my_dataset"
}

resource "google_bigquery_table" "table_for_view_testing" {
  dataset_id = google_bigquery_dataset.my_dataset.dataset_id
  table_id   = "private_table"

  # for now while we are testing
  deletion_protection = false

  time_partitioning {
    type = "DAY"
  }

  labels = {
    env = "default"
  }

  schema = <<EOF
[
  {
    "name": "requiredField",
    "type": "STRING",
    "mode": "REQUIRED",
    "description": "requiredField"
  },
  {
    "name": "optionalField",
    "type": "STRING",
    "mode": "NULLABLE",
    "description": "optionalField"
  }
]
EOF

}

resource "google_bigquery_table" "public_view_of_private_table" {
  dataset_id = google_bigquery_dataset.my_dataset.dataset_id
  table_id   = "public_view"

  # for now while we are testing
  deletion_protection = false

  time_partitioning {
    type = "DAY"
  }

  labels = {
    env = "default"
  }

  schema = <<EOF
[
  {
    "name": "requiredField",
    "type": "STRING",
    "mode": "REQUIRED",
    "description": "requiredField"
  },
  {
    "name": "optionalField",
    "type": "STRING",
    "mode": "NULLABLE",
    "description": "optionalField"
  }
]
EOF

  view {
    query = <<EOF
SELECT * FROM `<PUT_PROJECT_NAME_HERE>.my_dataset.private_table`
EOF
    use_legacy_sql = false
  }

}

Debug Output

Error: googleapi: Error 400: Provided Schema does not match Table wf-gcp-us-dserrao-sbx1-sbx:scribe_events.public_view. Field requiredField has changed mode from NULLABLE to REQUIRED, invalid
with google_bigquery_table.public_view_of_private_table
on view_not_working.tf line 37, in resource "google_bigquery_table" "public_view_of_private_table":
resource "google_bigquery_table" "public_view_of_private_table" {

Link to gist with terraform apply output as seen in Terraform-Enterprise https://gist.github.com/ds8080/1666d2446e09ab338803bdb9bdb7f7e1

Panic Output

Expected Behavior

VIew should have been created with schema matching the GBQ table

Actual Behavior

View gets created with mode:NULLABLE columns even though we specify it as mode:REQUIRED in the terraform. So the terraform apply fails

Steps to Reproduce

terraform apply

Important Factoids

References

0000

edwardmedia commented 1 year ago

I can repro this issue. The logic is to create a view without the schema first and then update the view with the schema. But the api does not allow changing the mode from NULLABLE to REQUIRED.

 POST /bigquery/v2/projects/myproject/datasets/issue14220/tables?alt=json&prettyPrint=false HTTP/1.1
 Host: bigquery.googleapis.com
 {
  "labels": {
   "env": "default"
  },
  "tableReference": {
   "datasetId": "issue14220",
   "projectId": "myproject",
   "tableId": "public_view"
  },
  "timePartitioning": {
   "type": "DAY"
  },
  "view": {
   "query": "SELECT * FROM `myproject.issue14220.private_table`\n",
   "useLegacySql": false
  }
 }

---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
{
"kind": "bigquery#table",
"etag": "lgwdPJC+OrywdLDBqT8/2g==",
"id": "myproject:issue14220.public_view",
"selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/myproject/datasets/issue14220/tables/public_view",
"tableReference": {
 "projectId": "myproject",
 "datasetId": "issue14220",
 "tableId": "public_view"
},
"labels": {
 "env": "default"
},
"schema": {
 "fields": [
  {
   "name": "requiredField",
   "type": "STRING",
   "mode": "NULLABLE"
  },
  {
   "name": "optionalField",
   "type": "STRING",
   "mode": "NULLABLE"
  }
 ]
},
"timePartitioning": {
 "type": "DAY"
},
"numBytes": "0",
"numLongTermBytes": "0",
"numRows": "0",
"creationTime": "1680732441241",
"lastModifiedTime": "1680732441578",
"type": "VIEW",
"view": {
  "query": "SELECT * FROM `myproject.issue14220.private_table`\n",
  "useLegacySql": false
 },
 "location": "US",
 "numTotalLogicalBytes": "0",
 "numActiveLogicalBytes": "0",
 "numLongTermLogicalBytes": "0"
}
-----------------------------------------------------

PUT /bigquery/v2/projects/myproject/datasets/issue14220/tables/public_view?alt=json&prettyPrint=false HTTP/1.1
Host: bigquery.googleapis.com
{
 "labels": {
  "env": "default"
 },
 "schema": {
  "fields": [
   {
    "description": "requiredField",
    "mode": "REQUIRED",
    "name": "requiredField",
    "type": "STRING"
   },
   {
    "description": "optionalField",
    "mode": "NULLABLE",
    "name": "optionalField",
    "type": "STRING"
   }
  ]
 },
 "tableReference": {
  "datasetId": "issue14220",
  "projectId": "myproject",
  "tableId": "public_view"
 },
 "timePartitioning": {
  "type": "DAY"
 },
 "view": {
  "query": "SELECT * FROM `myproject.issue14220.private_table`\n",
  "useLegacySql": false
 }
}

---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
{
  "error": {
    "code": 400,
    "message": "Provided Schema does not match Table myproject:issue14220.public_view. Field requiredField has changed mode from NULLABLE to REQUIRED",
    "errors": [
      {
        "message": "Provided Schema does not match Table myproject:issue14220.public_view. Field requiredField has changed mode from NULLABLE to REQUIRED",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

trodge commented 1 year ago

b/280326829

c2thorn commented 1 year ago

closed with https://github.com/GoogleCloudPlatform/magic-modules/pull/7973

This will release with v5.0.0. Check https://github.com/hashicorp/terraform-provider-google/issues/15582 for more details on the major release.

c2thorn commented 11 months ago

reopened due to revert: https://github.com/GoogleCloudPlatform/magic-modules/pull/9204

c2thorn commented 11 months ago

context: https://github.com/hashicorp/terraform-provider-google/issues/16134

github-actions[bot] commented 3 weeks ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-google