Impossible to create AlloyDB instance with PSC enabled

JPFrancoia commented 4 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.5.7 on darwin_arm64 (Mac OS, M1 machine)

provider registry.terraform.io/hashicorp/google 9e7dede23 (It's the main branch pulled this morning around 10 am UK time) on darwin_arm64 (Mac OS, M1 machine)

Affected Resource(s)

AlloyDB instance, possibly AlloyDB cluster

Terraform Configuration

resource "google_alloydb_cluster" "test_cluster" {
  cluster_id   = "test-cluster"
  location     = var.region
  database_version = "POSTGRES_15"

  initial_user {
    user     = "alloy_db_user"
    password = "password"
  }

  ## Works if provided instead of psc_config block
  #network_config {
    #network = google_service_networking_connection.private_vpc_connection.network
  #}

  psc_config {
    psc_enabled = true
  }
}

resource "google_alloydb_instance" "test_instance" {
  cluster       = google_alloydb_cluster.test_cluster.name
  instance_id   = "alloydb-instance"
  instance_type = "PRIMARY"

  database_flags = {
    "alloydb.logical_decoding" = "on" # Enable logical decoding for Datastream
  }

  machine_config {
    cpu_count = 2
  }

  #Same error with or without this block
  psc_instance_config {
      allowed_consumer_projects = ["979541276525"]
  }
}

terraform {
  required_providers {
    google = {
      source = "terraform.local/local/google"
      version = "5.32.0"
    }
  }
}

Debug Output

https://gist.github.com/JPFrancoia/6ab6ee3890e62eb6a3d1b816ece714b0

Expected Behavior

I expected an AlloyDB instance to be created in my AlloyDB cluster

Actual Behavior

The AlloyDB instance wasn't created, the terraform apply failed

Steps to reproduce

Build the main branch of this repo (commit 9e7dede23)

git clone https://github.com/hashicorp/terraform-provider-google.git
cd terraform-provider-google
go build -o terraform-provider-google
mkdir -p ~/.terraform.d/plugins/terraform.local/local/google/5.32.0/darwin_arm64
mv terraform-provider-google ~/.terraform.d/plugins/terraform.local/local/google/5.32.0/darwin_arm64

Then run terraform apply on the config I pasted earlier, and you should get the output pasted in the gist.

Important Factoids

For reasons that are out of scope for this issue, I need to create a Private Service Connect (PSC) endpoint for my AlloyDB database. The idea is to then use this endpoint to connect from a Cloud Run service. I'm aware that there are other solutions to connect CR and Alloy, but due to the network architecture of my company, I don't really have a choice here.

PSC isn't supported for Alloy on the most recent version of the provider (5.31.1 at the time of this issue). Luckily, it landed in main just yesterday: https://github.com/hashicorp/terraform-provider-google/pull/18263/files. Thanks to @vmiglani for that.

Their PR on the magic modules repo is here: https://github.com/GoogleCloudPlatform/magic-modules/pull/10783.

Unfortunately, I'm unable to get it to work.

Does anyone have an idea to unlock me? Really keen to support testing.

EDIT: Well I'm not sure it's actually a terraform provider bug. I tried to do the same operation with the gcloud CLI tool and hit an internal error too:

❯ gcloud --verbosity debug alloydb instances create alloydb-instance --instance-type=PRIMARY \
  --cpu-count=2 \
  --availability-type=REGIONAL \
  --region=europe-west2 \
  --cluster=test-cluster \
  --project=prj-vo-aa-s-dss-sandbox \
  --allowed-psc-projects=prj-vo-aa-s-dss-sandbox

DEBUG: Running [gcloud.alloydb.instances.create] with arguments: [--allowed-psc-projects: "['prj-vo-aa-s-dss-sandbox']", --availability-type: "REGIONAL", --cluster: "test-cluster", --cpu-count: "2", --instance-type: "PRIMARY", --project: "prj-vo-aa-s-dss-sandbox", --region: "europe-west2", --verbosity: "debug", INSTANCE: "alloydb-instance"]
DEBUG: Starting new HTTPS connection (1): alloydb.googleapis.com:443
DEBUG: https://alloydb.googleapis.com:443 "POST /v1/projects/prj-vo-aa-s-dss-sandbox/locations/europe-west2/clusters/test-cluster/instances?alt=json&instanceId=alloydb-instance HTTP/1.1" 200 None
Operation ID: operation-1717071341885-619aad3631653-55f84b44-89a79a20
Creating instance...⠶DEBUG: Starting new HTTPS connection (1): alloydb.googleapis.com:443
Creating instance...⠧DEBUG: https://alloydb.googleapis.com:443 "GET /v1/projects/prj-vo-aa-s-dss-sandbox/locations/europe-west2/operations/operation-1717071341885-619aad3631653-55f84b44-89a79a20?alt=json HTTP/1.1" 200 None
Creating instance...⠼DEBUG: Starting new HTTPS connection (1): alloydb.googleapis.com:443
DEBUG: https://alloydb.googleapis.com:443 "GET /v1/projects/prj-vo-aa-s-dss-sandbox/locations/europe-west2/operations/operation-1717071341885-619aad3631653-55f84b44-89a79a20?alt=json HTTP/1.1" 200 None
Creating instance...failed.
DEBUG: (gcloud.alloydb.instances.create) an internal error has occurred
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 998, in Execute
    resources = calliope_command.Run(cli=self, args=args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 815, in Run
    resources = command_instance.Run(args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/surface/alloydb/instances/create.py", line 120, in Run
    instance_operations.Await(op_ref, 'Creating instance',
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/api_lib/alloydb/instance_operations.py", line 49, in Await
    return waiter.WaitFor(
           ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 261, in WaitFor
    operation = PollUntilDone(
                ^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 322, in PollUntilDone
    operation = retryer.RetryOnResult(
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 249, in RetryOnResult
    if not should_retry(result, state):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 320, in _IsNotDone
    return not poller.IsDone(operation)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/google-cloud-sdk/477.0.0/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 122, in IsDone
    raise OperationError(operation.error.message)
googlecloudsdk.api_lib.util.waiter.OperationError: an internal error has occurred

References

No response

b/343741954

ggtisc commented 4 months ago

This looks like a new feature that isn't published yet on terraform registry or Google Cloud documentation and it isn't possible to reproduce.

I'm forwarding this for its validation

JPFrancoia commented 4 months ago

Yeah I think there is a bug in the Alloy DB API on Google's side, I don't think it's a terraform issue. If they fix the API this bug should be fixed too. This issue could still be useful though, if a release is made from the main branch now, anyone trying to deploy a cluster with PSC enabled will encounter this issue (until Google fixes the API).

vmiglani commented 4 months ago

This is an issue with the VPC-SC perimeter of this project due to their organization level VPC-SC setting. We are following up with the VPC-SC team to understand how this project can be unblocked from creating PSC-enabled AlloyDB clusters.

We have verified that AlloyDB cluster creation works correctly for projects that don't have VPC-SC restrictions. AlloyDB will be adding support for VPC-SC compliance in the near future.

hashicorp / terraform-provider-google