GCE instance failed to set metadata from workbench notebooks

[liusha-H]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.5.5

• provider registry.terraform.io/hashicorp/google v5.43.0 / v5.43.1

Affected Resource(s)

Workbench instance

Terraform Configuration

data "google_compute_network" "shared_vpc" {
  name    = var.vpc_name
  project = var.vpc_project_id
}

data "google_compute_subnetwork" "experiment_subnet" {
  name    = var.subnet_name
  region  = var.region
  project = var.vpc_project_id
}

resource "google_workbench_instance" "instances" {
  for_each        = { for instance in var.workbench_instances : instance.name => instance }
  name            = each.value.name
  location        = var.zone
  instance_owners = coalesce(each.value.instance_owners, [])
  project         = var.project

  gce_setup {
    machine_type = coalesce(each.value.machine_type, "e2-small")

    vm_image {
      project = try(each.value.vm_image.image_project, "deeplearning-platform-release")
      # name    = try(each.value.vm_image.image_name, "gcr.io/deeplearning-platform-release/sklearn-cpu")
      family = try(each.value.vm_image.image_family, "tf2-latest-cpu")
    }

    service_accounts {
      email = coalesce(
        each.value.service_account,
        "notebook-default@ki-datalake-experiments-7b57.iam.gserviceaccount.com"
      )
      #      scopes = [
      #        "https://www.googleapis.com/auth/cloud-platform",
      #        "https://www.googleapis.com/auth/userinfo.email",
      #        "https://www.googleapis.com/auth/bigquery"
      #      ]
    }

    boot_disk {
      disk_size_gb    = coalesce(each.value.boot_disk_size_gb, 50)
      disk_type       = coalesce(each.value.boot_disk_type, "PD_SSD")
      disk_encryption = coalesce(each.value.disk_encryption, "GMEK")
      # kms_key         = coalesce(each.value.disk_kms_key, "")
    }

    data_disks {
      disk_size_gb    = coalesce(each.value.data_disk_size_gb, 100)
      disk_type       = coalesce(each.value.boot_disk_type, "PD_SSD")
      disk_encryption = coalesce(each.value.disk_encryption, "GMEK")
      # kms_key         = coalesce(each.value.disk_kms_key, "")
    }

    network_interfaces {
      network = data.google_compute_network.shared_vpc.id
      subnet  = data.google_compute_subnetwork.experiment_subnet.id
    }

    metadata = {
      idle-timeout-seconds = "10800"
      terraform            = "true"
    }

    disable_public_ip    = true
    enable_ip_forwarding = false
    tags                 = coalesce(each.value.tags, [])

    dynamic "accelerator_configs" {
      for_each = each.value.accelerator_config != null ? ["fake"] : []
      content {
        type       = each.value.accelerator_config.type
        core_count = each.value.accelerator_config.core_count
      }
    }
  }

  disable_proxy_access = true

  labels = merge(
    module.labels.labels,
    each.value.labels
  )
}

Debug Output

Error: Error waiting for Updating Instance: Error code 3, message: failed updating GCE instance instance updater returned error: metadata key "version" is protected by Workbench Notebooks, invalid argument: invalid argument

 Error: Error waiting for Updating Instance: Error code 3, message: failed updating GCE instance instance updater returned error: metadata key "version" is protected by Workbench Notebooks, invalid argument: invalid argument

Expected Behavior

existing workbench instances updated and new instances created

Actual Behavior

missing argument as exit code 3

Steps to reproduce

1. terraform apply

Important Factoids

No response

References

No response

[ggtisc]

Hi @liusha-H !

I tried to replicate this issue, but you have a lot of data that we don't have access to. With the next configuration everything works fine without errors. You should check it, as well as each variable and value you have to be sure that your configuration is appropriate. If after this you still continue with issues share with us your full code to check your full code.

resource "google_service_account" "sa_19363" {
  account_id = "sa-19363"
}

resource "google_compute_network" "cn_19363" {
  name    = "cn-19363"
  auto_create_subnetworks = "false"
  project = "my-project"
}

resource "google_compute_subnetwork" "csn_19363" {
  name    = "csn-19363"
  network = google_compute_network.cn_19363.id
  ip_cidr_range = "10.0.36.0/24"
  region  = "us-central1"
  project = "my-project"
}

resource "google_workbench_instance" "wi_19363" {
  name            = "wi-19363"
  location        = "us-central1-a"
  instance_owners  = ["my-user@my-domain.com"]
  project = "my-project"

  gce_setup {
    machine_type = "n1-standard-4"

    vm_image {
      project = "cloud-notebooks-managed"
      family = "workbench-instances"
    }

    service_accounts {
      email = google_service_account.sa_19363.email
    }

      boot_disk {
          disk_size_gb  = 310
      disk_type = "PD_SSD"
      }

      data_disks {
          disk_size_gb  = 330
      disk_type = "PD_SSD"
      }

    network_interfaces {
      network = google_compute_network.cn_19363.id
      subnet = google_compute_subnetwork.csn_19363.id
    }

    metadata = {
      terraform = "true"
      idle-timeout-seconds = "10800"
    }

    disable_public_ip    = false
    enable_ip_forwarding = true
    tags = ["abc", "def"]

    accelerator_configs {
      type         = "NVIDIA_TESLA_T4"
      core_count   = 1
    }
  }

  disable_proxy_access = true

  labels = {
    k = "val"
  }

  desired_state = "ACTIVE"
}

[liusha-H]

Thanks @ggtisc I tried it last Friday, seems like it worked, not sure if that's a Google api issue maybe?

[liusha-H]

will double check today, see if it still the issue

[liusha-H]

Hi @ggtisc , just gave it a go this morning, and the issue seems fixed (only saw it happened between 2nd to 4th September), anyway, thanks for replying re this topic, feel free to close this issue please

[ggtisc]

It was a pleasure, have happy coding.