Concurrent activation of dependent APIs

bharathkkb commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

This maybe more of a question than an enhancement.

When activating multiple services in parallel using google_project_service, I have recently started running into errors like failed to send enable services request: googleapi: Error 400: Another activation or deactivation is in progress for the following service(s):foo. I have also observed that these go away if I de dupe the APIs which activate dependent APIs.

For instance

activate_apis = [
"container.googleapis.com",
"compute.googleapis.com",
]

may give an error as container.googleapis.com internally activates compute.googleapis.com. This puts the work on the user to figure out what are our dependent APIs and how to compose them. This becomes much harder with say for example anthos.googleapis.com which activates container.googleapis.com which activates compute.googleapis.com along with a dozen others.

New or Affected Resource(s)

google_project_service

Potential Terraform Configuration

N/A

References

b/164189639

edwardmedia commented 4 years ago

@bharathkkb I tried below code without hitting the error Error 400: Another activation or deactivation is in progress for the following service(s) Do you have other conditions that I should apply to repro it?

module "project_services" {
  source  = "terraform-google-modules/project-factory/google//modules/project_services"
  version = "3.3.0"
  project_id = "tf-test-issue7194"
  activate_apis = [
    "iam.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "clouddebugger.googleapis.com",
    "cloudfunctions.googleapis.com",
    "cloudidentity.googleapis.com",
    "cloudiot.googleapis.com",
    "cloudkms.googleapis.com",
    "cloudprofiler.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "cloudscheduler.googleapis.com",
    "cloudtrace.googleapis.com",
    "composer.googleapis.com",
    "compute.googleapis.com",
    "container.googleapis.com",
    "containerregistry.googleapis.com",
    "datacatalog.googleapis.com",
    "dataflow.googleapis.com",
    "dataproc.googleapis.com",
    "datastore.googleapis.com",
    "deploymentmanager.googleapis.com",
    "dns.googleapis.com",
    "fcm.googleapis.com",
    "firebase.googleapis.com",
    "firebasedynamiclinks.googleapis.com",
    "firebasehosting.googleapis.com",
    "firebaseinstallations.googleapis.com",
    "firebaseremoteconfig.googleapis.com",
    "firebaserules.googleapis.com",
    "firestore.googleapis.com",
    "gameservices.googleapis.com",
    "googlecloudmessaging.googleapis.com",
    "groupssettings.googleapis.com",
    "iam.googleapis.com",
    "iamcredentials.googleapis.com",
    "iap.googleapis.com",
    "identitytoolkit.googleapis.com",
    "language.googleapis.com",
    "logging.googleapis.com",
    "monitoring.googleapis.com",
    "networkmanagement.googleapis.com",
    "notebooks.googleapis.com",
    "osconfig.googleapis.com",
    "oslogin.googleapis.com",
    "pubsub.googleapis.com",
    "redis.googleapis.com",
    "resourceviews.googleapis.com",
    "run.googleapis.com",
    "runtimeconfig.googleapis.com",
    "secretmanager.googleapis.com",
  ]
  disable_services_on_destroy = false
  disable_dependent_services  = false
}

bharathkkb commented 4 years ago

@edwardmedia could you try with the anthos apis? Perhaps,

  activate_apis = [
    "container.googleapis.com",
    "compute.googleapis.com",
    "monitoring.googleapis.com",
    "logging.googleapis.com",
    "meshca.googleapis.com",
    "meshtelemetry.googleapis.com",
    "meshconfig.googleapis.com",
    "iamcredentials.googleapis.com",
    "anthos.googleapis.com",
    "gkeconnect.googleapis.com",
    "gkehub.googleapis.com",
    "cloudresourcemanager.googleapis.com",
  ]

In our case this has been sporadic and began on August 19th. From August 19th until Sept 2nd we saw this happen ~33 times out of ~100 builds and on 2nd we merged https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/pull/640 and yet to see this happen again. an example: https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/runs/1005472573

I will also try to repro this from my end.

edwardmedia commented 4 years ago

@bharathkkb Using your list, I have tried 5 times. All succeeded, without hitting issues.

bharathkkb commented 4 years ago

@edwardmedia thanks for investigating, I just opened a test PR https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/pull/657 to check from our side as well. This has debug logs enabled, so if we hit this issue, we should have some further info.

bharathkkb commented 4 years ago

@edwardmedia I was able to repro. My third test run https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/runs/1084084131 failed with this error. Debug logs attached to b/164189639

rileykarson commented 4 years ago

We should retry under these circumstances, beginning with the Sep 21 release (3.40.0, I think?).

bharathkkb commented 4 years ago

Thanks @rileykarson

ghost commented 3 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-google