Intermittent failures enabling bigquery via `google_project_service`

[davidstanke]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.0.7

Affected Resource(s)

google_project_service

Terraform Configuration Files

resource "google_project_service" "bq_api" {
  service                    = "bigquery.googleapis.com"
  disable_dependent_services = true
}

resource "google_bigquery_dataset" "four_keys" {
  dataset_id = "four_keys"
  location   = var.bigquery_region
  depends_on = [
    google_project_service.bq_api
  ]
}

Debug Output

https://gist.github.com/davidstanke/fb11edc13437a1b06f4f575cae7d6418

Expected Behavior

The dataset resource has an explicit dependency on the BiqQuery API resource, so the api should be enabled before TF attempts to create the dataset

Actual Behavior

TF was unable to create the dataset, because the API wasn't enabled. We have observed this behavior from time to time but not always. It's flaky! It appears that BQ sometimes returns a success status prior to being actually usable(?)

b/374161376

[melinath]

Are you able to confirm from the debug output that the google_project_service resource completes before the google_bigquery_dataset is attempted? If so, this is likely a race condition on the API side - see https://github.com/hashicorp/terraform-provider-google/issues/8938 for more details.

[davidstanke]

Hi @melinath, yes, we can confirm that Terraform reports that the google_project_service is complete before attempting to add the dataset, or to query it. So it appears that this is an API issue -- the API is reporting completion before it's actually ready to interact. How do we get this fixed?

[rileykarson]

Not a direct answer to your question about fixing it, but for additional Google-internal context also see b/228371710

[melinath]

@davidstanke one possible workaround would be to add a delay to your terraform plan.

[davidstanke]

Thanks @rileykarson and @melinath -- I've added a note to that internal bug and eagerly await a solution. Meanwhile, yeah: adding a delay may be the best option, though it makes me sad.