403 Error When Attempting To Enable Big Query With google_project_service(s)

mhorton-coalfire commented 5 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.12.7 terraform-provider-google_v2.14.0_x4

Affected Resource(s)

resource "google_project_services"

Terraform Configuration Files

# Define GCP Provider
provider "google" {
  credentials = var.credentials
}

# Define the backend used for remote state storage
terraform {
  backend "gcs" {}
  required_version = ">= 0.12.0"
}

# Project Resources

# Define project
resource "google_project" "project" {
  name                    = var.project
  project_id             = var.project
  billing_account     = var.billing_account
  folder_id               = var.folder

# Define enabled APIs within the project
resource "google_project_services" "project_services" {
  project         = google_project.project.project_id
  services        = [
    "cloudresourcemanager.googleapis.com",
    "cloudbilling.googleapis.com",
    "iam.googleapis.com",
    "compute.googleapis.com",
    "serviceusage.googleapis.com",
    "oslogin.googleapis.com",
    "cloudkms.googleapis.com",
    "storage-component.googleapis.com",
    "admin.googleapis.com",
    "dataproc.googleapis.com",
    "bigquery-json.googleapis.com",
    "bigquery.googleapis.com",
    "bigquerystorage.googleapis.com",
    "container.googleapis.com",
    "containerregistry.googleapis.com",
    "deploymentmanager.googleapis.com",
    "iamcredentials.googleapis.com",
    "pubsub.googleapis.com",
    "replicapool.googleapis.com",
    "replicapoolupdater.googleapis.com",
    "resourceviews.googleapis.com",
    "storage-api.googleapis.com",
  ]
  depends_on = ["google_project.project"]
}

# Define null provider
provider "null" {}

# Define delay command
resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "sleep 30"
  }
  triggers = {
    "project_services" = google_project_services.project_services.id
  }
}

Debug Output

Partial Output:

2019-09-04T18:03:22.337Z [DEBUG] plugin.terraform-provider-google_v2.14.0_x4: ----------------------------------------------------- 2019/09/04 18:03:22 [DEBUG] google_project_services.project_services: apply errored, but we're indicating that via the Error pointer rather than returning it: Error authoritatively enabling Project prj-wdm-analytics-cst-acme13 Services: unable to enable Project Services prj-wdm-analytics-cst-acme13 ([bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]): Batch "project/prj-wdm-analytics-cst-acme13/services:batchEnable" for request "Enable Project Services prj-wdm-analytics-cst-acme13: [bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]" returned error: failed to send enable services request: googleapi: Error 403: The caller does not have permission, forbidden 2019/09/04 18:03:22 [TRACE] : eval: terraform.EvalMaybeTainted 2019/09/04 18:03:22 [TRACE] EvalMaybeTainted: google_project_services.project_services encountered an error during creation, so it is now marked as tainted 2019/09/04 18:03:22 [TRACE] : eval: terrafopis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]" returned error: failed to send enable services request: googleapi: Error 403: The caller does not have permission, forbidden 2019/09/04 18:03:22 [ERROR] : eval: *terraform.EvalSequence, err: Error authoritatively enabling Project prj-wdm-analytics-cst-acme13 Services: unable to enable Project Services prj-wdm-analytics-cst-acme13 ([bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]): Batch "project/prj-wdm-analytics-cst-acme13/services:batchEnable" for request "Enable Project Services prj-wdm-analytics-cst-acme13: [bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis Error: Error authoritatively enabling Project prj-wdm-analytics-cst-acme13 Services: unable to enable Project Services prj-wdm-analytics-cst-acme13 ([bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]): Batch "project/prj-wdm-analytics-cst-acme13/services:batchEnable" for request "Enable Project Services prj-wdm-analytics-cst-acme13: [bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]" returned error: failed to send enable services request: googleapi: Error 403: The caller does not have permission, forbidden

Panic Output

N/A

Expected Behavior

The bigquery.googleapis.com API should be enabled.

Actual Behavior

The Big Query API fails to enable and produces the following error:

Error: Error authoritatively enabling Project prj-wdm-analytics-cst-acme13 Services: unable to enable Project Services prj-wdm-analytics-cst-acme13 ([bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]): Batch "project/prj-wdm-analytics-cst-acme13/services:batchEnable" for request "Enable Project Services prj-wdm-analytics-cst-acme13: [bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com dataproc.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com bigquery.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com admin.googleapis.com]" returned error: failed to send enable services request: googleapi: Error 403: The caller does not have permission, forbidden

Steps to Reproduce

Any attempt to enable bigquery.googleapis.com via terraform (google_project_services and well as google_project_service) fails with this 403 error (when applying). All other APIs are successfully enabled without any issue.

Important Factoids

My understanding is that the Big Query API is automatically enabled when a new project is created so this may be causing the conflict I am experiencing. This API enabling issue was discovered as a result of a separate failure in another of my modules. When attempting to assign the encrypter/decrypter role to the default big query service account: bq-[project number]@bigquery-encryption.iam.gserviceaccount.com" I receive an error that the service account does not exist. Therefore I am attempting to force its creation by enabling the API within terraform code. As of right now I have found no solution available within terraform that will result in the creation of the service account. Explicitly enabling the Big Query API is my last remaining hope.

References

None

JordanP commented 5 years ago

Just got the same issue with KMS API:

Error: Batch "project/xxx/services:batchEnable" for request "Enable Project Services xxx: [cloudkms.googleapis.com ]" returned error: failed to send enable services request: googleapi: Error 403: Service [cloudkms.googleapis.com ] not found or permission denied., forbidden

  on kms.tf line 1, in resource "google_project_service" "key_management_system":
   1: resource "google_project_service" "key_management_system" {

Terraform account is project owner, so should be fine

JordanP commented 5 years ago

So, it turns out I has an extra white space in the api name. I typed "cloudkms.googleapis.com ". Without this extra space, stuff works fine as expected :D

rileykarson commented 5 years ago

Out of curiosity @mhorton-coalfire, what indicated that you needed to enable bigquery.googleapis.com? I've used BigQuery in my project with just bigquery-json.googleapis.com I believe.

Based on @JordanP's experience, this may just be a case where the API returns a 403 instead of a 404 for not found.

mhorton-coalfire commented 5 years ago

@rileykarson This was brought about by an issue occurring in another one of my modules, which is that the default Big Query service account: bq-[project number]@bigquery-encryption.iam.gserviceaccount.com" does not exist for me to assign an encrypter/decrypter role for the Big Query specific KMS key I am creating when I apply. As a result the apply returns an error indicating the service account does not exist and the role cannot be assigned. My hope was that by enabling the generic API bigquery.googleapis.com I would force the creation of the default Big Query encryption service account (which apparently is not done when enabling the other two big query APIs in my code) and therefore solve my KMS role assignment problem. In attempting to test that solution I came across this issue with the bigquery.googleapis.com enabling.

rileykarson commented 5 years ago

@mhorton-coalfire: I don't believe that that service exists, and it's been aliased as bigquery-json instead. I'd suggest filing a bug against BigQuery itself that you don't have the service account in your project(s). I can't see it listed in their documentation, so I'm unsure why it may not exist.

Since there isn't anything Terraform can do here, I'm closing this out.

mhorton-coalfire commented 5 years ago

Will do. Thanks Riley!

Michael Horton CONSULTANT | CYBER ENGINEERING CCSK, GCP Professional Cloud Architect, AWS Solution Architect – Associate, AWS Security – Specialty, CompTIA Security+ michael.horton@coalfire.commailto:michael.horton@coalfire.com COALFIRE | Coalfire.comhttp://www.coalfire.com/ | (P) 720.501.4269 | (M) 405.664.6610 11000 Westmoor Circle | Suite 450 | Westminster | Colorado 80021 North America | Latin America | Europe How are we doing? Let us know at Feedback@Coalfire.commailto:Feedback@Coalfire.com

From: Riley Karson notifications@github.com Sent: Thursday, September 5, 2019 1:24 PM To: terraform-providers/terraform-provider-google terraform-provider-google@noreply.github.com Cc: Michael Horton Michael.Horton@Coalfire.com; Mention mention@noreply.github.com Subject: Re: [terraform-providers/terraform-provider-google] 403 Error When Attempting To Enable Big Query With google_project_service(s) (#4402)

@mhorton-coalfirehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mhorton-2Dcoalfire&d=DwMCaQ&c=w6l184r-L1wK6odsdmYZcA&r=TjvfixuvmSqZSPgAW1jyclnvn5q-3lpISrOwy7Pmh2Q&m=uBRUQlhLX6tQphynKwvzuo1epIMEaRYNtNxo54SbMZs&s=rwcv1w1eOLyxyX1FbsZMVlJbIzwxW5oF3o2vnV1_9Qw&e=: I don't believe that that service exists, and it's been aliased as bigquery-json instead. I'd suggest filing a bug against BigQuery itselfhttps://urldefense.proofpoint.com/v2/url?u=https-3A__issuetracker.google.com_issues_new-3Fcomponent-3D187149-26template-3D0&d=DwMCaQ&c=w6l184r-L1wK6odsdmYZcA&r=TjvfixuvmSqZSPgAW1jyclnvn5q-3lpISrOwy7Pmh2Q&m=uBRUQlhLX6tQphynKwvzuo1epIMEaRYNtNxo54SbMZs&s=7WAgdISe70T5QQrRuL-gY1YmlIyq6BJ_cAJU_VuA4Lg&e= that you don't have the service account in your project(s). I can't see it listed in their documentation, so I'm unsure why it may not exist.

Since there isn't anything Terraform can do here, I'm closing this out.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_terraform-2Dproviders_terraform-2Dprovider-2Dgoogle_issues_4402-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMTZYFTWMJISD6YLGIOBM4TQIFMGJA5CNFSM4ITVF4A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6AOO3Q-23issuecomment-2D528541550&d=DwMCaQ&c=w6l184r-L1wK6odsdmYZcA&r=TjvfixuvmSqZSPgAW1jyclnvn5q-3lpISrOwy7Pmh2Q&m=uBRUQlhLX6tQphynKwvzuo1epIMEaRYNtNxo54SbMZs&s=BR6XBPbCdDbzLm3LOhItO7WMwUcZnQBVpAsR4b9-NXU&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMTZYFV5QBGIA64BMXKPH2TQIFMGJANCNFSM4ITVF4AQ&d=DwMCaQ&c=w6l184r-L1wK6odsdmYZcA&r=TjvfixuvmSqZSPgAW1jyclnvn5q-3lpISrOwy7Pmh2Q&m=uBRUQlhLX6tQphynKwvzuo1epIMEaRYNtNxo54SbMZs&s=vkiHL8zS-p-ASmC7PhhbrwrZANBp0jiPV89YKuSqSeY&e=.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-google