Error with enabling services on a project

[morgante]

I'm currently seeing an odd condition where Terraform fails to activate services.

Terraform Version

Terraform v0.11.7
+ provider.google (master)
+ provider.gsuite (unversioned)
+ provider.null v1.0.0
+ provider.random v1.3.1

Affected Resource(s)

Please list the resources as a list, for example:

• google_project_service

Terraform Configuration Files

resource "google_project_service" "project_services" {
  count = "${length(var.activate_apis)}"

  project = "${local.project_id}"
  service = "${element(var.activate_apis, count.index)}"
}

Expected Behavior

It should activate the service.

Actual Behavior

I got a permission denied error.

* google_project_service.project_services[0]: 1 error(s) occurred:

* google_project_service.project_services.0: Error enabling service: Error enabling service ["compute.googleapis.com"] for project "rdc-exportservice-dev": Error waiting for api to enable: googleapi: Error 403: The caller does not have permission, forbidden

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

I verified that my service account does in fact have permission to activate services on that project:

$ gcloud services enable compute.googleapis.com --project=rdc-exportservice-dev 
Waiting for async operation operations/tmo-acf.2c0c9f47-04b0-43b1-ae32-94a83d7ea786 to complete...
Operation finished successfully. The following command can describe the Operation details:
 gcloud services operations describe operations/tmo-acf.2c0c9f47-04b0-43b1-ae32-94a83d7ea786

My Terraform service account owns the project.

[rosbo]

@danawillow, I think you were looking into some failures related to project services. Is this one related?

[danawillow]

It doesn't look like the exact same issue, but it's probably related. @morgante, do you have debug logs?

[morgante]

Thanks @danawillow, I don't have the old debug logs but after #1632 this is no longer happening.

[james-stephenson]

As of provider version 0.17.1, this still fails.

Repro:

terraform { backend "local" { path = "default.tfstate" } }
provider "google" { region = "us-east-4" }

variable "project_id" { default = "services-fail-xxx" }

variable "project_services" {
  default = [
    "cloudresourcemanager.googleapis.com",
    "cloudbilling.googleapis.com",
    "iam.googleapis.com",
    "compute.googleapis.com",
    "oslogin.googleapis.com",
    "storage-api.googleapis.com",
    "serviceusage.googleapis.com",
  ]
}

resource "google_project" "project" {
  name = "Google Services Won't Work"
  project_id = "${var.project_id}"

  folder_id="folders/xxx"
  billing_account="xxx"
}

resource "google_project_services" "services" {
  project     = "${var.project_id}"
  services    = "${var.project_services}"
}

Output:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + google_project.project
      id:                  <computed>
      auto_create_network: "true"
      billing_account:     "xxx"
      folder_id:           "xxx"
      name:                "Google Services Won't Work"
      number:              <computed>
      org_id:              <computed>
      policy_data:         <computed>
      policy_etag:         <computed>
      project_id:          "services-fail-xxx"
      skip_delete:         <computed>

  + google_project_services.services
      id:                  <computed>
      disable_on_destroy:  "true"
      project:             "services-fail-xxx"
      services.#:          "7"
      services.1560437671: "iam.googleapis.com"
      services.1568433289: "oslogin.googleapis.com"
      services.1954675454: "serviceusage.googleapis.com"
      services.2240314979: "compute.googleapis.com"
      services.3644083179: "cloudresourcemanager.googleapis.com"
      services.3875785048: "storage-api.googleapis.com"
      services.3902838863: "cloudbilling.googleapis.com"

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_project_services.services: Creating...
  disable_on_destroy:  "" => "true"
  project:             "" => "services-fail-xxx"
  services.#:          "" => "7"
  services.1560437671: "" => "iam.googleapis.com"
  services.1568433289: "" => "oslogin.googleapis.com"
  services.1954675454: "" => "serviceusage.googleapis.com"
  services.2240314979: "" => "compute.googleapis.com"
  services.3644083179: "" => "cloudresourcemanager.googleapis.com"
  services.3875785048: "" => "storage-api.googleapis.com"
  services.3902838863: "" => "cloudbilling.googleapis.com"
google_project.project: Creating...
  auto_create_network: "" => "true"
  billing_account:     "" => "xxx"
  folder_id:           "" => "xxx"
  name:                "" => "Google Services Won't Work"
  number:              "" => "<computed>"
  org_id:              "" => "<computed>"
  policy_data:         "" => "<computed>"
  policy_etag:         "" => "<computed>"
  project_id:          "" => "services-fail-xxx"
  skip_delete:         "" => "<computed>"
google_project.project: Still creating... (10s elapsed)
google_project.project: Creation complete after 14s (ID: services-fail-xxx)

Error: Error applying plan:

1 error(s) occurred:

* google_project_services.services: 1 error(s) occurred:

* google_project_services.services: Error creating services: failed to list services: googleapi: Error 403: The caller does not have permission, forbidden

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[danawillow]

Hey @james-stephenson, a few questions:

• Are you authenticating using a service account or using your gcloud credentials?
• Did this same config work in older versions of Terraform, or are you coming to it fresh?
• Do you have debug logs? Those are always useful for us to be able to help you.

[james-stephenson]

It look like you must add the depends_on block in order to make this work, which, I assume, wasn't intended. Terraform will not properly wait for the project to be created before continuing on to the services resource unless you add it.

This might actually be a problem with dependency resolution, whereby this resource isn't waiting for the project to complete creation.

[james-stephenson]

That code above was the simplest reproduction I could create to show the problem. I am testing using my gcloud default authentication credentials, though I believe this also fails with a service account.

If you look at the output I pasted above, it's clearly not waiting for the project to exist, and even tries creating the services before it tries to create the project.

[morgante]

Yeah, your config doesn't tell Terraform that the project should be created before the services. You can resolve that by adding a depends_on or, even better, using a reference to create the implicit dependency.

resource "google_project_services" "services" {
  project     = "${google_project.project.project_id}"
  services    = "${var.project_services}"
}

[james-stephenson]

Ugh, that's right. Sorry for the false alarm :/

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!