Migrating to google_project_service from google_project_services = much slower

Stono commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

0.12.18, with 3.3.0 google plugin

Affected Resource(s)

google_project_service

Terraform Configuration Files


locals {
  admin_enabled_apis = [
    "bigquerystorage.googleapis.com",
    "bigquery-json.googleapis.com",
    "storage-component.googleapis.com",
    "cloudtrace.googleapis.com",
    "containerregistry.googleapis.com",
    "pubsub.googleapis.com",
    "compute.googleapis.com",
    "servicemanagement.googleapis.com",
    "dns.googleapis.com",
    "deploymentmanager.googleapis.com",
    "replicapool.googleapis.com",
    "replicapoolupdater.googleapis.com",
    "resourceviews.googleapis.com",
    "monitoring.googleapis.com",
    "logging.googleapis.com",
    "container.googleapis.com",
    "storage-api.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "iam.googleapis.com",
    "cloudbilling.googleapis.com",
    "cloudkms.googleapis.com",
    "serviceusage.googleapis.com",
    "oslogin.googleapis.com",
    "stackdriver.googleapis.com",
    "sqladmin.googleapis.com",
    "redis.googleapis.com",
    "iamcredentials.googleapis.com",
    "dataflow.googleapis.com"
  ]
}

resource "google_project_service" "enabled-apis" {
  for_each = toset(local.admin_enabled_apis)
  service  = each.value
  disable_dependent_services = false
  disable_on_destroy         = true
}

Debug Output

Panic Output

Expected Behavior

google_project_service implementation to be as fast as google_project_services was.

Actual Behavior

Since google_project_services was deprecated, we now have to declare a google_project_services for each api we enable in terraform. We have some 40 apis used across 15 different projects, as a result a terraform plan now takes, absolutely ages.

I can only presume that previously, google_project_services retrieved the list of all enabled apis for the project in one call, whereas now that's happening for each google_project_service.

This is that bad, we're likely going to stop using terraform to track google_project_service, which isn't great - as we'd like everything in git as code.

Steps to Reproduce

terraform apply

Important Factoids

References

0000

Stono commented 4 years ago

reading into this more from a similar issue (https://github.com/terraform-providers/terraform-provider-google/issues/4689), it appears that batching is limited by parallelism, which defaults to 10. I'm able to get my plan time down by setting this to 100, but that feels like a strange workaround and will likely cause me to hit other api rate limits.

emilymye commented 4 years ago

We actually also batch calls to get the project services - this might actually be due to the fact that our batches are created on intervals, and it defaults to 10 seconds. We have a parameter for setting this interval here in the provider configuration:
https://www.terraform.io/docs/providers/google/guides/provider_reference.html#send_after - could you try seeing if reducing this interval helps your plan time?

We should probably also look into reducing this default time, given the initial limit on number of concurrent requests is pretty low.

Stono commented 4 years ago

Hey, I set the timer to 3 seconds, combined with setting parallelism to 100 and my plan time is now 1m 33s, whereas on the old provider it was 3m 12s, so it's actually faster :-)

Thanks for your help!

emilymye commented 4 years ago

Glad to hear it :) I'll go ahead and close the issue now!

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-google