Unable to Create Apigee X Subscription Org

chrisohiou57 commented 5 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

Terraform Version: 1.7.5 Google Provider Version: 5.22.0 Google Beta Provider Version: 5.22.0

Affected Resource(s)

google_apigee_organization

Terraform Configuration

data-sources.tf

data "google_compute_network" "apigee_network" {
  project = var.shared_vpc_host_project_id
  name = var.apigee_network_name
}

data "google_kms_key_ring" "kms-org-db" {
  project  = var.apigee_service_project_id
  name     = var.apigee_org_key_ring_name
  location = var.apigee_org_key_ring_location
}

data "google_kms_crypto_key" "kms-key-org-db" {
  name     = var.apigee_org_crypto_key_name
  key_ring = data.google_kms_key_ring.kms-org-db.id
}

main.tf

resource "google_apigee_organization" "apigee_x" {
  project_id                           = var.apigee_service_project_id
  analytics_region                     = var.analytics_region
  authorized_network                   = data.google_compute_network.apigee_network.id
  runtime_database_encryption_key_name = data.google_kms_crypto_key.kms-key-org-db.id
  billing_type                         = var.billing_type
  retention                            = var.retention
  runtime_type                         = "CLOUD"

  properties {
    # This empty block and providing the runtime_type was needed to get rid of a WARNing:
    # [WARN]  Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for google_apigee_organization.apigee_x, but we are tolerating it because it is using the legacy plugin SDK.
    # The following problems may be the cause of any confusing errors from downstream operations:
    # - .properties: attribute representing nested block must not be unknown itself; set nested attribute values to unknown instead google_apigee_organization.apigee_x
  }
}

variables.tfvars

shared_vpc_host_project_id   = "shared-vpc-host-project-id"
apigee_service_project_id    = "shared-vpc-service-project-id-with-apigee-entitlements-enabled"
apigee_network_name          = "prod"
analytics_region             = "us-central1"
apigee_org_key_ring_name     = "apigee-prod-org"
apigee_org_key_ring_location = "us"
apigee_org_crypto_key_name   = "org-database"
billing_type                 = "SUBSCRIPTION"
retention                    = "MINIMUM"

Debug Output

Terraform says the plan is:

  # google_apigee_organization.apigee_x will be created
  + resource "google_apigee_organization" "apigee_x" {
      + analytics_region                     = "us-central1"
      + apigee_project_id                    = (known after apply)
      + authorized_network                   = "projects/shared-vpc-host-395501/global/networks/prod"
      + billing_type                         = (known after apply)
      + ca_certificate                       = (known after apply)
      + id                                   = (known after apply)
      + name                                 = (known after apply)
      + project_id                           = "apigee-prod-394115"
      + retention                            = "MINIMUM"
      + runtime_database_encryption_key_name = "projects/apigee-prod-394115/locations/us/keyRings/apigee-prod-org/cryptoKeys/org-database"
      + runtime_type                         = "CLOUD"
      + subscription_type                    = (known after apply)

      + properties {
        }
    }

2024-03-29T14:27:43.095-0500 [TRACE] terraform.contextPlugins: Schema for provider "registry.terraform.io/hashicorp/google" is in the global cache 2024-03-29T14:27:43.095-0500 [DEBUG] google_apigee_organization.apigee_x: applying the planned Create change 2024-03-29T14:27:43.095-0500 [TRACE] GRPCProvider: ApplyResourceChange 2024-03-29T14:27:43.095-0500 [TRACE] GRPCProvider: GetProviderSchema 2024-03-29T14:27:43.095-0500 [TRACE] GRPCProvider: returning cached schema: EXTRA_VALUE_AT_END=registry.terraform.io/hashicorp/google 2024-03-29T14:27:43.096-0500 [TRACE] provider.terraform-provider-google_v5.22.0_x5: Received request: @module=sdk.proto tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=e09859eb-048a-e398-1704-95a4c407f24e tf_resource_type=google_apigee_organization @caller=github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/tf5server/server.go:846 tf_rpc=ApplyResourceChange timestamp=2024-03-29T14:27:43.096-0500 2024-03-29T14:27:43.096-0500 [TRACE] provider.terraform-provider-google_v5.22.0_x5: Sending request downstream: tf_rpc=ApplyResourceChange tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=e09859eb-048a-e398-1704-95a4c407f24e @caller=github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/internal/tf5serverlogging/downstream_request.go:20 @module=sdk.proto tf_resource_type=google_apigee_organization timestamp=2024-03-29T14:27:43.096-0500 2024-03-29T14:27:43.096-0500 [TRACE] provider.terraform-provider-google_v5.22.0_x5: calling downstream server: tf_mux_provider="schema.GRPCProviderServer" tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-mux@v0.13.0/internal/logging/mux.go:19 @module=sdk.mux timestamp=2024-03-29T14:27:43.096-0500 2024-03-29T14:27:43.096-0500 [TRACE] provider.terraform-provider-google_v5.22.0_x5: Calling downstream: @module=sdk.helper_schema tf_resource_type=google_apigee_organization tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.31.0/helper/schema/resource.go:908 tf_req_id=e09859eb-048a-e398-1704-95a4c407f24e tf_mux_provider="schema.GRPCProviderServer" tf_provider_addr=registry.terraform.io/hashicorp/google timestamp=2024-03-29T14:27:43.096-0500 2024-03-29T14:27:43.096-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Creating new Organization: map[string]interface {}{"analyticsRegion":"us-central1", "authorizedNetwork":"projects/shared-vpc-host-project-id/global/networks/prod", "name":"shared-vpc-service-project-id-with-apigee-entitlements-enabled", "runtimeDatabaseEncryptionKeyName":"projects/shared-vpc-service-project-id-with-apigee-entitlements-enabled/locations/us/keyRings/apigee-prod-org/cryptoKeys/org-database", "runtimeType":"CLOUD"} 2024-03-29T14:27:43.096-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Waiting for state to become: [success] 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Retry Transport: starting RoundTrip retry loop 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Retry Transport: request attempt 0 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Google API Request Details: 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: ---[ REQUEST ]--------------------------------------- 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: POST /v1/organizations?alt=json&parent=projects%2Fshared-vpc-service-project-id-with-apigee-entitlements-enabled HTTP/1.1 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Host: apigee.googleapis.com 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: User-Agent: Terraform/1.7.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.31.0 terraform-provider-google/5.22.0 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Content-Length: 286 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Content-Type: application/json 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Accept-Encoding: gzip 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: { 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "analyticsRegion": "us-central1", 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "authorizedNetwork": "projects/shared-vpc-host-project-id/global/networks/prod", 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "name": "shared-vpc-service-project-id-with-apigee-entitlements-enabled", 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "runtimeDatabaseEncryptionKeyName": "projects/shared-vpc-service-project-id-with-apigee-entitlements-enabled/locations/us/keyRings/apigee-prod-org/cryptoKeys/org-database", 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "runtimeType": "CLOUD" 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: } 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5 2024-03-29T14:27:43.097-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: ----------------------------------------------------- 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Google API Response Details: 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: ---[ RESPONSE ]-------------------------------------- 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: HTTP/2.0 400 Bad Request 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Cache-Control: private 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Content-Type: application/json; charset=UTF-8 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Date: Fri, 29 Mar 2024 19:27:43 GMT 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Server: ESF 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Vary: Origin 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Vary: X-Origin 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Vary: Referer 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: X-Content-Type-Options: nosniff 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: X-Frame-Options: SAMEORIGIN 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: X-Xss-Protection: 0 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: { 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "error": { 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "code": 400, 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "message": "Request contains an invalid argument.", 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "status": "INVALID_ARGUMENT", 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "details": [ 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: { 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "@type": "type.googleapis.com/google.rpc.RequestInfo", 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "requestId": "12622108079478799005" 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: } 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: ] 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: } 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: } 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: ----------------------------------------------------- 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 400 with body: HTTP/2.0 400 Bad Request 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Cache-Control: private 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Content-Type: application/json; charset=UTF-8 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Date: Fri, 29 Mar 2024 19:27:43 GMT 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Server: ESF 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Vary: Origin 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Vary: X-Origin 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: Vary: Referer 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: X-Content-Type-Options: nosniff 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: X-Frame-Options: SAMEORIGIN 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: X-Xss-Protection: 0 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: { 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "error": { 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "code": 400, 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "message": "Request contains an invalid argument.", 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "status": "INVALID_ARGUMENT", 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "details": [ 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: { 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "@type": "type.googleapis.com/google.rpc.RequestInfo", 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: "requestId": "12622108079478799005" 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: } 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: ] 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: } 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: } 2024-03-29T14:27:43.366-0500 [DEBUG] provider.terraform-provider-google_v5.22.0_x5: 2024/03/29 14:27:43 [DEBUG] Retry Transport: Returning after 1 attempts

Expected Behavior

The Apigee X subscription org is created.

Actual Behavior

We always get a 400 Bad Request Invalid Parameter when Terraform attempts to create the Apigee organization.

Steps to reproduce

terraform apply --var-file variables.tfvars

Important Factoids

It seems some of the documented arguments for the resource are not added to the Google API request even though the values show up in the plan.

We tried various configurations:

We tried the google-beta provider but got the same result.
We tried removing optional arguments (e.g., billing_type, runtime_type, etc.)
We deleted the .terraform/providers directory so that we got a fresh download of the provider dependencies.

I did a find and replace on the Shared VPC host and service project IDs. That is the only thing that is different than the actual log on the build machine.

My apologies in advance if we're doing something obviously silly, but we spent a fair amount of time looking at the docs and playing with the config. We also went through the efforts to update everything to the latest version on the build machine.

References

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/apigee_organization https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/apigee_organization

b/332376079

slevenick commented 5 months ago

Let me start by saying that the error message you're receiving is really unhelpful. It makes it really difficult to determine what is going on, so I understand that this is a bad experience.

I do believe that the Terraform resource works in general, as we have tests that pass that are based off of the examples shown in the documentation.

If you start with just the fields set in that example, can you create the organization successfully? Maybe it's something about the values you've chosen that are for some reason disallowed by the API? We can ask someone from the service team to chime in here, but it may take some time

chrisohiou57 commented 5 months ago

Thank you for the quick reply @slevenick!

We just tried again with the simplest config that we can use. We have a requirement to use a CMK. Here's what we ran. Unfortunately, we're still getting the same error.

resource "google_apigee_organization" "apigee_x" {
  project_id                           = var.apigee_service_project_id
  analytics_region                     = var.analytics_region
  authorized_network                   = data.google_compute_network.apigee_network.id
  runtime_database_encryption_key_name = data.google_kms_crypto_key.kms-key-org-db.id
}

We compared the plan value for runtime_database_encryption_key_name with the crypto key resource name that you can copy out of the web console and they match. The service account we're using for Terraform has the Compute Network User role on the shared vpc host project. The prod network that we're using has one subnet for us-central1 that has been peered with the servicenetowrking VPC.

We may pivot to using apigeecli or gcloud if this will take a while.

slevenick commented 5 months ago

Yeah, unfortunately I have no way to tell what is causing this. We'll need the Apigee team themselves to chime in

hashicorp / terraform-provider-google