Variable transformation on google_workbench_instance leads to Bad Request (HTTP 400)

caetano-colin commented 5 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

Terraform v1.7.5

Affected Resource(s)

google_workbench_instance

Terraform Configuration

resource "google_workbench_instance" "instance" {
  disable_proxy_access = false
  instance_owners = []
  labels = {
    "boundry_code" = "001"
    "environment"  = "development"
  }
  location = "us-central1-a"
  name     = "test-ml-instance"
  project  = "prj-d-bu3machine-learning-ma6i"
  gce_setup {
    disable_public_ip    = true
    machine_type         = "e2-standard-4"
    metadata = {
      "disable-mixer"              = "false"
      "notebook-disable-downloads" = "true"
      "notebook-disable-root"      = "true"
      "notebook-disable-terminal"  = "true"
      "notebook-upgrade-schedule"  = "00 19 * * MON"
      "report-dns-resolution"      = "true"
      "report-event-health"        = "true"
      "terraform"                  = "true"
    }
    tags = [
      "egress-internet",
    ]
    boot_disk {
      disk_encryption = "CMEK"
      disk_size_gb    = "150"
      disk_type       = "PD_SSD"
      kms_key         = "projects/prj-d-kms-3i3k/locations/us-central1/keyRings/sample-keyring/cryptoKeys/prj-d-bu3machine-learning"
    }
    data_disks {
      disk_encryption = "CMEK"
      disk_size_gb    = "150"
      disk_type       = "PD_SSD"
      kms_key         = "projects/prj-d-kms-3i3k/locations/us-central1/keyRings/sample-keyring/cryptoKeys/prj-d-bu3machine-learning"
    }
    network_interfaces {
      #network = "https://www.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/567715592102256480"
      #network = "https://www.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted"
      network = "projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted"
      subnet  = "projects/prj-d-shared-restricted-9enw/regions/us-central1/subnetworks/sb-d-shared-restricted-us-central1"
    }
    vm_image {
      family  = "workbench-instances"
      project = "cloud-notebooks-managed"
    }
  }

Debug Output

https://gist.github.com/caetano-colin/8435956b49b1eb761ac51653f1727f58

Expected Behavior

The workbench_instance should deploy succesfully using the specified network/subnetwork set.

Actual Behavior

I received a 400 bad request and the input format changed from projects/{projectId}/global/networks/{network_id} to an URL.

It seems like the URL is not an expected field according to this document: https://cloud.google.com/vertex-ai/docs/workbench/reference/rest/v2/projects.locations.instances#NetworkInterface

Steps to reproduce

terraform apply

Important Factoids

Calling this API endpoint using curl, with the same parameters as the terraform debug output but just changing the body networkInterface.network format from the URL to the expected format, projects/{projectId}/global/networks/{network_id}, works fine
The Google Cloud Console uses the format above and also works fine, it seems like when using an URL to specify the network I always get a 400 (Bad Request)

References

https://cloud.google.com/vertex-ai/docs/workbench/reference/rest/v2/projects.locations.instances#NetworkInterface

I'm not 100% sure, but it seems like this is the part of the provider source code responsible for transforming the variable to an URL:

func expandWorkbenchInstanceGceSetupNetworkInterfacesNetwork(v interface{}, d tpgresource.TerraformResourceData, config *transport_tpg.Config) (interface{}, error) {
    if v == nil || v.(string) == "" {
        return "", nil
    } else if strings.HasPrefix(v.(string), "https://") {
        return v, nil
    }
    url, err := tpgresource.ReplaceVars(d, config, "{{ComputeBasePath}}"+v.(string))
    if err != nil {
        return "", err
    }
    return tpgresource.ConvertSelfLinkToV1(url), nil
}

b/338447520

ggtisc commented 5 months ago

Hi @caetano-colin

This error indicates that something with the network and subnet is wrong, and you can confirm it in the response message that you are sharing:

"message": "value_to_check(https://compute.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted) **is not found**."

I suggest you to check those resources and see if they exists and their configuration(zone, location, project, roles, permissions, and so on)

caetano-colin commented 5 months ago

Hello @ggtisc, thank you for your response!

I can confirm that the resources do indeed exist. As mentioned earlier in the issue, the identical request succeeds when sent via curl or the console (using projects/{projectId}/global/networks/{network_id} format). However, the problem arises specifically when the request is made through the terraform provider, on the execution of "expandWorkbenchInstanceGceSetupNetworkInterfacesNetwork" function, this appears to alter the format in a manner that leads to a bad request (from projects/{projectId}/global/networks/{network_id} to https://compute.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted).

Just to prove that this network exists, here is the output of the curl https://compute.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted -H "Authorization: Bearer $(gcloud auth print-access-token)" command:

{
  "kind": "compute#network",
  "id": "567715592102256480",
  "creationTimestamp": "2024-04-12T04:57:03.926-07:00",
  "name": "vpc-d-shared-restricted",
  "selfLink": "https://www.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted",
  "selfLinkWithId": "https://www.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/567715592102256480",
  "autoCreateSubnetworks": false,
  "subnetworks": [
    "https://www.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/regions/us-central1/subnetworks/sb-d-shared-restricted-us-central1",
    "https://www.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/regions/us-west1/subnetworks/sb-d-shared-restricted-us-west1"
  ],
  "peerings": [
    ...
  ],
  "routingConfig": {
    ...
  },
  "networkFirewallPolicyEnforcementOrder": "AFTER_CLASSIC_FIREWALL"
}

ggtisc commented 5 months ago

This is effectively as the official documentation you shared describes. There aren't another current way to reference the network and subnetwork for the attribute network interfaces of the google_workbench_instance

caetano-colin commented 5 months ago

Upon reviewing the debug output linked below at line 40, We can see that the network attribute underwent transformation into a URL before being sent to the Google API Server:

https://gist.github.com/caetano-colin/8435956b49b1eb761ac51653f1727f58#file-gistfile1-txt-L40

From projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted to https://compute.googleapis.com/compute/v1/projects/prj-d-shared-restricted-9enw/global/networks/vpc-d-shared-restricted

My hypothesis is that this transformation is causing the issue, preventing me from creating the resource through the Terraform provider.

The behaviour I observed on my tests before opening this issue was:

If I exactly replicate the request captured in the debug output and manually remove the https://compute.googleapis.com/compute/v1 prefix from the network parameter in the request body, the resource is created successfully.

ggtisc commented 5 months ago

This is an internal harassment, but according to the official documentation the only thing you need to do is declare the resources as they are on these links:

This ticket was replicated again from scratch with the provided version without errors

caetano-colin commented 5 months ago

Yes it does work on most of the times, for example, I was able to create the same instance in a different project and using the same network.

The odd thing is that the provider does change the format of the parameter to an undocumented format, and in the project mentioned above, I get the error.

Being able to create this resource with the same parameters but using a manual curl request to the same endpoint or through the Google Cloud Console instead of using the Terraform Provider, strongly suggests that this issue is not a misconfiguration issue but a provider issue.

Here are more details about the experiment I made:

https://gist.github.com/caetano-colin/02e66f73af472e6a6f9c141e577a65af

Please don't hesitate to reach out if there's anything further I can assist you with regarding this matter. I'm fully committed to providing my best effort to help resolve this problem.

ggtisc commented 5 months ago

In this case it can be a proposal for a future implementation. But for now the unique available configurations are those

renato-rudnicki commented 4 months ago

I`m also facing the same issue.

mariammartins commented 4 months ago

same problem here

github-actions[bot] commented 2 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-google