terraform wait for eventarc service agent readiness

[mbychkowski]

It's a common issue with the Service Agent provisioning system, which requires us to only create the Service Agent Service Account on the first use. It also has a very long propagation delay (the communicated SLO is up to 7 minutes). This causes initial deployment to fail at event arc trigger creation because of agent readiness.

• As a workaround, we can manually call services.GenerateServiceIdentity API to provision the P4SA and then add a sleep for a couple of minutes before resuming the process. The Terraform docs are here: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_service_identity. GCP API docs are here https://cloud.google.com/service-usage/docs/reference/rest/v1beta1/services/generateServiceIdentity