GCP instance VM creation issue

yogiswar28 commented 1 year ago

Community Note

When creating VM instances with CMEK is failing to create stating Constraint constraints/gcp.restrictNonCmekServices violated for project attempting to create a resource not protected by CMEK crypto key.
GCP project is having Organisation policy which is having constraint constraints/gcp.restrictNonCmekServices.

Terraform Version

v1.3.7

Affected Resource(s)

google_compute_instance

Terraform Configuration Files

resource "google_compute_disk" "boot" { project = var.project zone = var.zone

snapshot = var.name name = var.name type = "pd-ssd" size = "12"

labels = var.labels disk_encryption_key { kms_key_self_link = var.cmek } }

resource "google_compute_disk" "data" { project = var.project zone = var.zone

snapshot = var.name name = var.name type = "pd-ssd" size = "100"

labels = var.labels disk_encryption_key { kms_key_self_link = var.cmek } }

resource "google_compute_instance" "fortigate" { project = var.project zone = var.zone deletion_protection = true

machine_type = "n2-standard-8" name = var.name

metadata = { enable-oslogin = false }

boot_disk { source = google_compute_disk.boot.self_link auto_delete = false }

attached_disk { source = google_compute_disk.data.self_link }

labels = var.labels tags = var.tags

dynamic "network_interface" { for_each = var.networks content { network = network_interface.value["network"] subnetwork = network_interface.value["subnetwork"] network_ip = network_interface.value["ip_address"] } }

service_account { email = var.service_account scopes = var.service_account_scopes }

allow_stopping_for_update = true can_ip_forward = true }

Expected Behavior

VM instance has to be created with the CMEK key.

Actual Behavior

VM is not creating with below mentioned error.
Constraint constraints/gcp.restrictNonCmekServices violated for project attempting to create a resource not protected by CMEK crypto key

Steps to Reproduce

Update the above code with respective details and run the below commands.

terraform init
terraform plan
terraform apply Below is the debug log debug.txt

yogiswar28 commented 1 year ago

Here is the GCP Rest API code: POST https://www.googleapis.com/compute/v1/projects//zones/europe-west3-c/instances { "canIpForward": false, "confidentialInstanceConfig": { "enableConfidentialCompute": false }, "deletionProtection": false, "description": "", "disks": [ { "autoDelete": true, "boot": true, "deviceName": "test-instance", "diskEncryptionKey": { "kmsKeyName": "projects//locations/europe-west3/keyRings/key-ring-unbound/cryptoKeys/lz-kms-key" }, "initializeParams": { "diskSizeGb": "10", "diskType": "projects//zones/europe-west3-c/diskTypes/pd-balanced", "labels": {}, "sourceImage": "projects/debian-cloud/global/images/debian-11-bullseye-v20230306" }, "mode": "READ_WRITE", "type": "PERSISTENT" } ], "displayDevice": { "enableDisplay": false }, "guestAccelerators": [], "keyRevocationActionType": "NONE", "labels": { "ec-src": "vm_add-rest" }, "machineType": "projects//zones/europe-west3-c/machineTypes/e2-medium", "metadata": { "items": [ { "key": "enable-oslogin", "value": "true" } ] }, "name": "test-instance", "networkInterfaces": [ { "stackType": "IPV4_ONLY", "subnetwork": "projects//regions/europe-west3/subnetworks/-subnet" } ], "params": { "resourceManagerTags": {} }, "reservationAffinity": { "consumeReservationType": "ANY_RESERVATION" }, "scheduling": { "automaticRestart": true, "onHostMaintenance": "MIGRATE", "provisioningModel": "STANDARD" }, "serviceAccounts": [ { "email": "sa-fortigate@project.iam.gserviceaccount.com", "scopes": [ "https://www.googleapis.com/auth/cloud-platform" ] } ], "shieldedInstanceConfig": { "enableIntegrityMonitoring": true, "enableSecureBoot": false, "enableVtpm": true }, "tags": { "items": [] }, "zone": "projects//zones/europe-west3-c" }

Here instance, disks are part of compute api and both will be creating at a time.

edwardmedia commented 1 year ago

@yogiswar28 looks like it is blocked by your org policy. Maybe you want to check with who owns the policy?

search by constraints/gcp.restrictNonCmekServices

https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints

yogiswar28 commented 1 year ago

Yes, we have enabled the org policy as per the requirement. Even we have the policy enabled we are able to create from the GCP console via gcloud/restapi. When we are trying to create from terraform it is creating only the disks and not the instance.

As per GCP disks and vm instance part of compute API. We can only pass the cmek to disks and not to the instance.

Is there any option to enable the cmek at instance level or it should be consider disk cmek to the instance too.

edwardmedia commented 1 year ago

@yogiswar28 to be honest, I don't know what went wrong. The error directly complains the violation of below constraint. After you have the policy disabled, do you still receive the same error?

Constraint constraints/gcp.restrictNonCmekServices violated for project <project_id> attempting to create a resource not protected by CMEK crypto key.

Maybe also review https://cloud.google.com/kms/docs/cmek-org-policy

yogiswar28 commented 1 year ago

Hello,

Yes after enabled the policy disks were creating without any problem as I have added key. But its not creating the instance, which is searching for a key while creating the instance.

What I understand from gcp api is instance and disks are part of compute api. We can pass the key details to disks but there is no option for instance creation.

If I exempt the policy for compute api then no use of using the cmek policy.

What i want to understand is if there is chance that it will consider disk key also as instance key or do i need to specificy the key while creating instance. If yes then how can I pass the key to instance via terraform.

Regards, Yogiswar

On Fri, 7 Apr, 2023, 5:13 pm Edward Sun, @.***> wrote:

@yogiswar28 https://github.com/yogiswar28 to be honest, I don't know what went wrong. The error directly complains the violation of below constraint. After you have the policy enabled, do you still receive the same error?

Constraint constraints/gcp.restrictNonCmekServices violated for project attempting to create a resource not protected by CMEK crypto key.

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/terraform-provider-google/issues/14189#issuecomment-1500378074, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5Z6YFLPDWCRFIAPD7KKEJTXAAVJ3ANCNFSM6AAAAAAWSG3SRM . You are receiving this because you were mentioned.Message ID: @.***>

edwardmedia commented 1 year ago

@SarahFrench any idea what went wrong?

yogiswar28 commented 1 year ago

Hello

Do you have any fix for this or any one tried with the same scenario.

Regards.

SarahFrench commented 1 year ago

Hi @yogiswar28 - sorry for the delay, I'm trying to get access to a GCP project with the same policy constraints to test this out.

You said:

What i want to understand is if there is chance that it will consider disk key also as instance key or do i need to specificy the key while creating instance. If yes then how can I pass the key to instance via terraform.

I think that the constraint is being triggered when creating the instance because no CMEKs are included in the POST to create the instance. From your debug.txt attachment I can see that in the POST request that triggers the error the diskEncryptionKey field isn't present, which specifies the CMEK key that the VM needs to use the disks. Absence of this field might be what is triggering the constraints/gcp.restrictNonCmekServices constraint, but I'm unable to test this at the moment.

Could you please test changing your configuration to include the disk_encryption_key_raw argument in your google_compute_instance configuration block?

Like this:

resource "google_compute_instance" "fortigate" {
  project             = var.project
  zone                = var.zone
  deletion_protection = true

  machine_type = "n2-standard-8"
  name         = var.name

  metadata = {
    enable-oslogin = false
  }

  boot_disk {
    source      = google_compute_disk.boot.self_link
    disk_encryption_key_raw = var.cmek # new argument added
    auto_delete = false
  }

  attached_disk {
    source = google_compute_disk.data.self_link
    disk_encryption_key_raw = var.cmek # new argument added
  }

   [other arguments omitted]
}

If you test this, could you please include a new debug log if there's still an error. In the meantime I'll continue trying to get access to a test environment with the same organisational constraint so I can manually test this too.

yogiswar28 commented 1 year ago

Thanks for the inputs. I will check and get back.

trodge commented 9 months ago

@yogiswar28 did the apply succeed with the new configuration?

hashicorp / terraform-provider-google