Enable Service Usage API (serviceusage.googleapis.com) by default when creating new projects

[verbanicm]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Description

After creating a project no APIs will be enabled. If the user_project_override config is true and no billing_project is given, subsequent requests to list APIs via the serviceusage API will fail. This is caused by the list services request using the newly created project.ProjectId as the X-Goog-User-Project creating an impossible to resolve situation without manually enabling the serviceusage API.

By default the Google Cloud console enables a set of service that is too broad to enable all by default: https://cloud.google.com/service-usage/docs/enabled-service#default

For this use case it makes sense to always enable the Service Usage API as it is required to list and enable any other services after creating a project.

Terraform Version

v1.3.6

Affected Resource(s)

• google_project

Terraform Configuration Files

This is modified from my configuration...

resource "google_project" "test" {
  folder_id  = google_folder.default.name
  project_id = "some-unique-project-id"

  name            = "some-unique-project-id"
}

resource "google_project_service" "test" {
  project = google_project.test.project_id

  service                    = "serviceusage.googleapis.com"
  disable_on_destroy         = false
  disable_dependent_services = false
}

terraform {
  required_version = ">= 1.0"

  required_providers {
    google = {
      version = ">= 4.45"
    }
  }
}

provider "google" {
  user_project_override = true
}

Debug Output

Error: Error when reading or editing Project Service : Request `List Project Services XXXXXXXXXX` returned error: Batch request 
  and retried single request "List Project Services XXXXXXXXXX" both failed. Final error: Failed to list enabled services for project
  XXXXXXXXXX: googleapi: Error 403: Service Usage API has not been used in project XXXXXXXXXX before or it is disabled. Enable it by
  visiting https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=XXXXXXXXXX then retry. If you
  enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Google developers console API activation",
        "url": "https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=XXXXXXXXXX"
      }
    ]
  },
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/XXXXXXXXXX",
      "service": "serviceusage.googleapis.com"
    },
    "reason": "SERVICE_DISABLED"
  }
]
, accessNotConfigured

Expected Behavior

Project is created and serviceusage.googleapis.com is enabled.

Actual Behavior

I get an error scenario where the service usage api can list itself on the project so it is unable to turn it on.

Steps to Reproduce

1. terraform apply

Important Factoids

• user_project_override = true
• No billing_project is specified, ideally to use the target resources project

[rileykarson]

This is a known interaction, I'm not sure if we'd want to modify the google_project resource to enable services as proposed in https://github.com/hashicorp/terraform-provider-google/pull/14173- particularly in a minor release. Will respond in more depth when I have cycles, unblocking our release right now.

[rileykarson]

Alright, took me a bit longer to get back than expected. Sorry about that! Also fyi @c2thorn in case you have input.

There are numerous reasons we likely want to avoid making this change, in my opinion:

• Enabling a service requires a different set of permissions and quotas than creating a project
• If the service enablement call fails, the project resource will be tainted. Taints are hard to interact with for many users who don't run Terraform directly & interactively, and are a rough state for a difficult-to-delete resource like a project to enter.
  • https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project#auto_create_network is conceptually similar- it does a service enablement and makes a GCE API call. Failures manifested differently when this was implemented and tainting did not occur. I do not believe we would implement this today. Instead we'd take a subresource/fine-grained resource approach like with https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_default_service_accounts. A fine-grained resource for project services would be no different than google_project_service.
• The Console and gcloud enable different projects by default, but Terraform's a different tool with a different UX. Given it's relatively low level- somewhere between client libraries and gcloud- having behaviour closer to the API is expected. It would be surprising to end users if we enabled additional services given some customers use enabled services to control what features their developers are allowed to use. Google [centrally] has not taken a strong stance on service activation as a security measure.
• A change in the default provider behaviour is a breaking change. Guarding this behind a flag would largely defeat the purpose, as a user would need to understand the issue and set the right flag anyways- they could just resolve the issue directly in not much more time.

This is caused by the list services request using the newly created project.ProjectId as the X-Goog-User-Project creating an impossible to resolve situation without manually enabling the serviceusage API.

It's not impossible to resolve- this can be resolved by using a billing project or by using an aliased provider without UPO enabled.

---

I'll also note that the impact here is likely fairly small. It's relatively uncommon that a configuration both needs UPO set and is creating a project:

• If you're using credentials other than ADCs UPO almost never needs to be enabled
• Larger customers typically use a "seed project" model where projects and services are provisioned centrally- both operations that don't need UPO enabled- while other resources are provisioned in application projects where it's fine to enable UPO
• Most services don't need UPO to interact with them, even for ADCs

[rileykarson]

Note: We discussed this. We're somewhat undecided- this would help new users and users with complex ticket-based provisioning systems, but tainting the project in the case of a failure is a pretty rough outcome.

[roaks3]

I'm not sure if this issue should be forwarded to the service team in its current state, so exempting it for now.

[jean-mercier-hivebrite]

Same problem i didn't get what is the workaround sorry ?

today we enable the API by hand everytime we need a new project but it's not something perfect ...

[TakumiHaruta]

Same here. Does anyone have the workaround for this issue? I believe it's almost synonymous with not being able to create and manage projects from Terraform if we can't solve this issue...