Add support for `schema` in bigquery_job.load

[colincadams]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Add support for the schema field for bigquery load jobs.

https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationLoad

This field is helpful if you cannot rely on the schema being autodetected properly. Without this field, there is no way to override the schema for WRITE_TRUNCATE jobs. Setting the schema on the table itself doesn't help, as it is overwritten by the job.

New or Affected Resource(s)

• google_bigquery_job

Potential Terraform Configuration

resource "google_storage_bucket_object" "table_data" {
  name   = "${var.table_name}.csv"
  bucket = var.bucket_name
  source = "${var.table_name}.csv"
}

resource "google_bigquery_table" "table" {
  dataset_id = var.dataset_id
  table_id   = var.table_name

  schema = var.schema
}

resource "google_bigquery_job" "load" {
  job_id = "${var.table_name}_load_${md5(google_storage_bucket_object.table_data.crc32c)}"

  load {
    source_uris = [
      "gs://${var.bucket_name}/${google_storage_bucket_object.table_data.output_name}"
    ]

    schema = var.schema

    destination_table {
      project_id = google_bigquery_table.table.project
      dataset_id = google_bigquery_table.table.dataset_id
      table_id   = google_bigquery_table.table.table_id
    }

    skip_leading_rows = 1

    write_disposition = "WRITE_TRUNCATE"
  }
}

References

As a start, I pushed what I think would be the necessary change here: https://github.com/GoogleCloudPlatform/magic-modules/compare/master...colincadams:colincadams/add-schema-bq-load

The configuration is copied from google_bigquery_table, save for the description which I pulled from the API docs.

b/374161354

[ggtisc]

Hi @colincadams!

Currently there are many options to manage this like:

• copy.write_dsposition
• load.schema_update_options
• load.write_disposition
• schema_update_options
• write_disposition