Issue with airbyte_destination_bigquery_denormalized resource using GCS staging loading method

[krazavet-tinyclues]

Hello 👋

We are facing an issue while attempting to create our Airbyte stack through Terraform. The problem arises when using the airbyte_destination_bigquery_denormalized resource with the GCS staging loading method for BigQuery. We have provided the relevant code snippet below:

resource "airbyte_destination_bigquery_denormalized" "this" {
  name         = format("shopify-bigquery-denorm-%s", var.environment)
  workspace_id = airbyte_workspace.this.workspace_id

  configuration = {
    project_id                      = var.project_id
    dataset_id                      = "shopify_denormalized"
    dataset_location                = "US"
    destination_type                = "bigquery-denormalized"
    credentials_json                = base64decode(google_service_account_key.airbyte.private_key)
    big_query_client_buffer_size_mb = 15
    loading_method = {
      destination_bigquery_denormalized_loading_method_gcs_staging = {
        method          = "GCS Staging"
        gcs_bucket_name = google_storage_bucket.airbyte.name
        gcs_bucket_path = "gcs_staging_denormalized"
        credential = {
          destination_bigquery_denormalized_loading_method_gcs_staging_credential_hmac_key = {
            credential_type    = "HMAC_KEY"
            hmac_key_access_id = google_storage_hmac_key.airbyte.access_id
            hmac_key_secret    = google_storage_hmac_key.airbyte.secret
          }
        }
        file_buffer_count = 10
      }
    }
  }
}

The specific error we encounter is as follows:

module.airbyte.airbyte_destination_bigquery_denormalized.this: Creating...
╷
│ Error: unexpected response from API. Got an unexpected response code 422
│
│   with module.airbyte.airbyte_destination_bigquery_denormalized.this,
│   on ../../../../modules/airbyte/airbyte.tf line 5, in resource "airbyte_destination_bigquery_denormalized" "this":
│    5: resource "airbyte_destination_bigquery_denormalized" "this" {
│
│ **Request**:
│ POST /v1/destinations HTTP/1.1
│ Host: [api.airbyte.com](http://api.airbyte.com/)
│ Accept: application/json
│ Authorization: Bearer
│ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
│ Content-Type: application/json
│ User-Agent: speakeasy-sdk/terraform 0.1.2 internal 1.0.0
│
│
│ **Response**:
│ HTTP/2.0 422 Unprocessable Entity
│ Content-Length: 348
│ Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
│ Content-Type: application/problem+json
│ Date: Fri, 7 Jul 2023 14:49:01 GMT
│ Server: envoy
│ Via: 1.1 google
│ X-Content-Type-Options: nosniff
│ X-Frame-Options: SAMEORIGIN
│ X-Xss-Protection: 0
│
│ {"type":"https://reference.airbyte.com/reference/errors#unprocessable-entity","title":"unprocessable-entity","status":422,"detail":"The
│ provided configuration does not fulfill the specification. Errors: json schema validation failed when comparing the data
│ to the json schema. \nErrors: $.loading_method.method: must be a constant value Standard "}

We attempted an alternative loading configuration method as shown below:

resource "airbyte_destination_bigquery_denormalized" "this" {
  name         = format("shopify-bigquery-denorm-%s", var.environment)
  workspace_id = airbyte_workspace.this.workspace_id

  configuration = {
    project_id                      = var.project_id
    dataset_id                      = "shopify_denormalized"
    dataset_location                = "US"
    destination_type                = "bigquery-denormalized"
    credentials_json                = base64decode(google_service_account_key.airbyte.private_key)
    big_query_client_buffer_size_mb = 15
    loading_method = {
      destination_bigquery_denormalized_update_loading_method_gcs_staging = {
        method          = "GCS Staging"
        gcs_bucket_name = google_storage_bucket.airbyte.name
        gcs_bucket_path = "gcs_staging_denormalized"
        credential = {
          destination_bigquery_denormalized_update_loading_method_gcs_staging_credential_hmac_key = {
            credential_type    = "HMAC_KEY"
            hmac_key_access_id = google_storage_hmac_key.airbyte.access_id
            hmac_key_secret    = google_storage_hmac_key.airbyte.secret
          }
        }
        file_buffer_count = 10
      }
    }
  }
}

With this alternative configuration, we were able to create the resource. However, when checking the Airbyte UI, the GCS staging loading method did not work as expected.

It would be greatly appreciated if you could provide a valid example of how to configure the airbyte_destination_bigquery_denormalized resource with the GCS staging loading method. We had difficulties understanding the documentation on the Terraform website, and having some examples would be very helpful.

[krazavet-tinyclues]

Hello 👋

Any news regarding this issue 🙏 Maybe @rileybrook ? Sorry I mentioned you as you answered on a previous issue 😅

I would like to understand how to configure a BigQuery denormalized destination with GCS loading method.

Best regards, Kevin

[rileybrook]

@airbytehq/platform-enablement Can we update our terraform docs with an accurate code example for creating a BigQuery destination? References 1, 2, 3 🙏