RE filter not work with 'data "google_compute_instance_template"'

Varrkan82 commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.4.4 on linux_amd64

Affected Resource(s)

google_compute_instance_template

Terraform Configuration Files


module "instance_template" {
  source                       = "./.terraform/modules/vm/modules/instance_template"
  for_each                     = toset(var.name_prefix)
  name_prefix                  = each.key
  service_account              = var.service_account
  disk_size_gb                 = each.key == "k8s-master" ? "50" : "100"
  auto_delete                  = false
  preemptible                  = false
  project_id                   = var.project_id
  enable_nested_virtualization = true
  machine_type                 = each.key == "k8s-worker" ? var.machine_type : "n1-standard-2"
  source_image_family          = var.source_image_family
  source_image_project         = var.source_image_project
  source_image                 = var.source_image
  tags                         = [each.key, "node"]
  network                      = module.vpc.network_self_link
  subnetwork                   = each.key == "k8s-master" ? module.vpc.subnets_self_links[0] : module.vpc.subnets_self_links[1]
  can_ip_forward               = true
  startup_script               = each.key == "k8s-master" ? local.k8s_master : local.k8s_worker
}

data "google_compute_instance_template" "master_link" {
  project     = var.project_id
  filter      = "name ~ k8s-master-"
  most_recent = true
  depends_on = [
    module.instance_template
  ]
}

data "google_compute_instance_template" "worker_link" {
  project     = var.project_id
  filter      = "name ~ k8s-worker-"
  most_recent = true
  depends_on = [
    module.instance_template
  ]
}

output "template_master" {
  value = data.google_compute_instance_template.master_link.self_link
}

output "template_worker" {
  value = data.google_compute_instance_template.worker_link.self_link
}

Debug Output


│ Error: error retrieving list of instance templates: googleapi: Error 400: Invalid value for field 'filter': 'name ~ k8s-master-'. Invalid list filter expression., invalid
│ 
│   with data.google_compute_instance_template.master_link,
│   on main.tf line 181, in data "google_compute_instance_template" "master_link":
│  181: data "google_compute_instance_template" "master_link" {
│ 

│ Error: error retrieving list of instance templates: googleapi: Error 400: Invalid value for field 'filter': 'name ~ k8s-worker-'. Invalid list filter expression., invalid
│ 
│   with data.google_compute_instance_template.worker_link,
│   on main.tf line 190, in data "google_compute_instance_template" "worker_link":
│  190: data "google_compute_instance_template" "worker_link" {
│ 

[$] <git:(master*)> gcloud compute instance-templates list --filter="name ~ k8s-master-"
NAME                                   MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP
k8s-master-20230410145352429800000001  n1-standard-2               2023-04-10T09:30:02.860-07:00
[$] <git:(master*)> gcloud compute instance-templates list --filter="name ~ k8s-worker-"
NAME                                   MACHINE_TYPE  PREEMPTIBLE  CREATION_TIMESTAMP
k8s-worker-20230410145353392600000002  e2-micro                   2023-04-10T09:30:03.636-07:00

Expected Behavior

Should be returned some result like it returned in gcloud compute instance-templates list --filter="name ~ k8s-master-"

Actual Behavior

Filter validation failed at a Plan state.

Steps to Reproduce

terraform plan

b/308756419

Varrkan82 commented 1 year ago

I thought it is related to Terraform data filter and created issue there, but they pushed me back from Terraform repository to this provider repo...

I've also tried to filter output for data "google_compute_image" and got the same result.

Plan: 0 to add, 1 to change, 0 to destroy.

│ Error: error retrieving list of images: googleapi: Error 400: Invalid value for field 'filter': 'name ~ debian'. Invalid list filter expression., invalid
│ 
│   with data.google_compute_image.name,
│   on main.tf line 181, in data "google_compute_image" "name":
│  181: data "google_compute_image" "name" {
│ 
[$] <git:(master*)> gcloud compute images list --filter="name ~ debian-"     
NAME                                PROJECT       FAMILY           DEPRECATED  STATUS
debian-10-buster-v20230306          debian-cloud  debian-10                    READY
debian-11-bullseye-arm64-v20230306  debian-cloud  debian-11-arm64              READY
debian-11-bullseye-v20230306        debian-cloud  debian-11                    READY

edwardmedia commented 1 year ago

@Varrkan82 not sure what you have in the module. But does below data sources working for you?

data "google_compute_image" "my_image" {
  family  = "debian-11"
  project = "debian-cloud"
}

data "google_compute_instance_template" "generic-regex" {
  filter      = "name != generic-tpl-20200107"
  most_recent = true
}

Varrkan82 commented 1 year ago

@edwardmedia yes. This works almost correct and returned the next output. Just added a project to the last block as it generates an access error without it.

data "google_compute_instance_template" "generic-regex" {
  project = var.project_id
  filter      = "name != generic-tpl-20200107"
  most_recent = true
}

Output:

Changes to Outputs:
  + cloud_image      = {
      + archive_size_bytes                = 1679224320
      + creation_timestamp                = "2023-03-06T12:57:17.193-08:00"
      + description                       = "Debian, Debian GNU/Linux, 11 (bullseye), amd64 built on 20230306, supports Shielded VM features"
      + disk_size_gb                      = 10
      + family                            = "debian-11"
      + filter                            = null
      + id                                = "projects/debian-cloud/global/images/debian-11-bullseye-v20230306"
      + image_encryption_key_sha256       = ""
      + image_id                          = "7157337275614620995"
      + label_fingerprint                 = "42WmSpB8rSM="
      + labels                            = {}
      + licenses                          = [
          + "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/licenses/debian-11-bullseye",
        ]
      + name                              = "debian-11-bullseye-v20230306"
      + project                           = "debian-cloud"
      + self_link                         = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-11-bullseye-v20230306"
      + source_disk                       = ""
      + source_disk_encryption_key_sha256 = ""
      + source_disk_id                    = ""
      + source_image_id                   = ""
      + status                            = "READY"
    }
  + regex            = {
      + advanced_machine_features    = null
      + can_ip_forward               = false
      + confidential_instance_config = null
      + description                  = ""
      + disk                         = [
          + {
              + auto_delete                    = true
              + boot                           = true
              + device_name                    = "persistent-disk-0"
              + disk_encryption_key            = []
              + disk_name                      = ""
              + disk_size_gb                   = 100
              + disk_type                      = "pd-standard"
              + interface                      = "SCSI"
              + labels                         = {}
              + mode                           = "READ_WRITE"
              + resource_policies              = []
              + source                         = ""
              + source_image                   = "projects/debian-cloud/global/images/family/debian-11"
              + source_image_encryption_key    = []
              + source_snapshot                = ""
              + source_snapshot_encryption_key = []
              + type                           = "PERSISTENT"
            },
        ]
      + filter                       = "name != generic-tpl-20200107"
      + guest_accelerator            = null
      + id                           = "projects/weblate-383021/global/instanceTemplates/bastion-instance-template-20230410195907346800000001"
      + instance_description         = ""
      + labels                       = null
      + machine_type                 = "e2-micro"
      + metadata                     = {
          + enable-oslogin = "TRUE"
          + ssh-keys       = <<-EOT
                vbieliavtsev_gmail_com:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJFeKzC4XEF0l9J2DiWafyT1xqFSRhhAZE18DaWwFHGd v-key
            EOT
        }
      + metadata_fingerprint         = "xqK0q07uDy8="
      + metadata_startup_script      = null
      + min_cpu_platform             = ""
      + most_recent                  = true
      + name                         = "bastion-instance-template-20230410195907346800000001"
      + name_prefix                  = null
      + network_interface            = [
          + {
              + access_config      = []
              + alias_ip_range     = []
              + ipv6_access_config = []
              + ipv6_access_type   = ""
              + name               = "nic0"
              + network            = "https://www.googleapis.com/compute/v1/projects/weblate-383021/global/networks/k8s-vpc"
              + network_ip         = ""
              + nic_type           = ""
              + queue_count        = 0
              + stack_type         = ""
              + subnetwork         = "https://www.googleapis.com/compute/v1/projects/weblate-383021/regions/us-east1/subnetworks/vm-subnet"
              + subnetwork_project = "weblate-383021"
            },
        ]
      + project                      = "weblate-383021"
      + region                       = "us-east1"
      + reservation_affinity         = null
      + resource_policies            = null
      + scheduling                   = [
          + {
              + automatic_restart           = true
              + instance_termination_action = ""
              + min_node_cpus               = 0
              + node_affinities             = []
              + on_host_maintenance         = "MIGRATE"
              + preemptible                 = false
              + provisioning_model          = "STANDARD"
            },
        ]
      + self_link                    = "https://www.googleapis.com/compute/v1/projects/weblate-383021/global/instanceTemplates/bastion-instance-template-20230410195907346800000001"
      + self_link_unique             = "https://www.googleapis.com/compute/v1/projects/weblate-383021/global/instanceTemplates/bastion-instance-template-20230410195907346800000001?uniqueId=5567541374218782075"
      + service_account              = [
          + {
              + email  = "bastion@weblate-383021.iam.gserviceaccount.com"
              + scopes = [
                  + "https://www.googleapis.com/auth/cloud-platform",
                ]
            },
        ]
      + shielded_instance_config     = [
          + {
              + enable_integrity_monitoring = true
              + enable_secure_boot          = true
              + enable_vtpm                 = true
            },
        ]
      + tags                         = []
      + tags_fingerprint             = ""
    }

Here I don't see all of my templates. Only the first one bastion. If I filtered excluding bastion image with filter = "name != bastion-instance-template-20230410195907346800000001" - there printed a single one template with name k8s-worker-20230410145353392600000002. There present one more with name k8s-master-20230410145352429800000001 an it not displayed in an output.

Varrkan82 commented 1 year ago

Update: filter like filter = "name:k8s-master-*" works fine for me.

edwardmedia commented 1 year ago

@Varrkan82 Glad you found a solution. Is there still pending issues?

Varrkan82 commented 1 year ago

Is there still pending issues?

I think this issue still pending as a RegExp filtering not works as expected and documented here

filter - (Optional) A filter to retrieve the instance templates. See gcloud topic filters for reference. If multiple instance templates match, either adjust the filter or specify most_recent. One of name, filter or self_link_unique must be provided.

And as it works with gcloud tool.

edwardmedia commented 1 year ago

@Varrkan82 do you have an example of using RegExp filtering that does not work for terraform but works for gcloud?

Varrkan82 commented 1 year ago

@Varrkan82 do you have an example of using RegExp filtering that does not work for terraform but works for gcloud?

Yes. I've added it above. Please look to issue body and the first my comment to it. I've tried a different ways to filter out: like ^k8s-master-.* or .*k8s-master-.*. But as I i think - this should have an exactly the same behaviour as a tool. And gcloud successfully filtering with name ~ k8s-master- filter.

edwardmedia commented 1 year ago

@Varrkan82 I got 0 items by running below command. Did you say you got items back? If yes, mind sharing the output of your execution by appending --log-http?

gcloud compute instance-templates list --filter="name ~ k8s-master-"  --log-http

edwardmedia commented 1 year ago

@Varrkan82 is this still an issue?

Varrkan82 commented 1 year ago

gcloud compute instance-templates list --filter="name ~ k8s-master-" --log-http

Here is an output with log-http enabled

$ gcloud compute instance-templates list --filter="name ~ k8s-master-" --log-http
=======================
==== request start ====
uri: https://oauth2.googleapis.com/token
method: POST
== headers start ==
b'content-type': b'application/x-www-form-urlencoded'
b'user-agent': b'google-cloud-sdk gcloud/427.0.0 command/gcloud.compute.instance-templates.list invocation-id/7d9f61*******************5ef environment/None environment-version/None client-os/LINUX client-os-ver/5.15.90 client-pltf-arch/x86_64 interactive/True from-script/False python/3.9.16 term/xterm-256color (Linux 5.15.90.1-microsoft-standard-WSL2)'
== headers end ==
== body start ==
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this request.
== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Encoding: gzip
Content-Type: application/json; charset=utf-8
Date: Wed, 26 Apr 2023 13:33:12 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: scaffolding on HTTPServer2
Transfer-Encoding: chunked
Vary: Origin, X-Origin, Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 0
-- headers end --
-- body start --
Body redacted: Contains oauth token. Set log_http_redact_token property to false to print the body of this response.
-- body end --
total round trip time (request+response): 0.932 secs
---- response end ----
----------------------
=======================
==== request start ====
uri: https://compute.googleapis.com/compute/v1/projects/weblate-383021/global/instanceTemplates?alt=json&filter=name+eq+%22.%2A%28k8s-master-%29.%2A%22&maxResults=500
method: GET
== headers start ==
b'accept': b'application/json'
b'accept-encoding': b'gzip, deflate'
b'authorization': --- Token Redacted ---
b'content-length': b'0'
b'user-agent': b'google-cloud-sdk gcloud/427.0.0 command/gcloud.compute.instance-templates.list invocation-id/7d9f618*****************835ef environment/None environment-version/None client-os/LINUX client-os-ver/5.15.90 client-pltf-arch/x86_64 interactive/True from-script/False python/3.9.16 term/xterm-256color (Linux 5.15.90.1-microsoft-standard-WSL2)'
== headers end ==
== body start ==

== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Encoding: gzip
Content-Type: application/json; charset=UTF-8
Date: Wed, 26 Apr 2023 13:33:13 GMT
ETag: PPPzL**********************************************6AyyE=
Server: ESF
Transfer-Encoding: chunked
Vary: Origin, X-Origin, Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 0
-- headers end --
-- body start --
{
  "kind": "compute#instanceTemplateList",
  "id": "projects/weblate-383021/global/instanceTemplates",
  "items": [
    {
      "kind": "compute#instanceTemplate",
      "id": "29840125943126071",
      "creationTimestamp": "2023-04-26T06:27:52.580-07:00",
      "name": "k8s-master-20230426115237974800000002",
      "description": "",
      "properties": {
        "tags": {
          "items": [
            "k8s-master",
            "node"
          ]
        },
        "machineType": "n1-standard-2",
        "canIpForward": true,
        "networkInterfaces": [
          {
            "kind": "compute#networkInterface",
            "network": "https://www.googleapis.com/compute/v1/projects/weblate-383021/global/networks/vm-net",
            "subnetwork": "https://www.googleapis.com/compute/v1/projects/weblate-383021/regions/us-east1/subnetworks/vm-subnet",
            "name": "nic0"
          }
        ],
        "disks": [
          {
            "kind": "compute#attachedDisk",
            "type": "PERSISTENT",
            "mode": "READ_WRITE",
            "deviceName": "persistent-disk-0",
            "index": 0,
            "boot": true,
            "initializeParams": {
              "sourceImage": "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230302",
              "diskSizeGb": "100",
              "diskType": "pd-standard"
            },
            "interface": "SCSI"
          }
        ],
        "serviceAccounts": [
          {
            "email": "v***********@weblate-383021.iam.gserviceaccount.com",
            "scopes": [
              "https://www.googleapis.com/auth/source.read_only",
              "https://www.googleapis.com/auth/compute",
              "https://www.googleapis.com/auth/logging.write",
              "https://www.googleapis.com/auth/monitoring",
              "https://www.googleapis.com/auth/servicecontrol",
              "https://www.googleapis.com/auth/service.management.readonly",
              "https://www.googleapis.com/auth/devstorage.read_only"
            ]
          }
        ],
        "scheduling": {
          "onHostMaintenance": "MIGRATE",
          "automaticRestart": true,
          "preemptible": false,
          "provisioningModel": "STANDARD"
        },
        "advancedMachineFeatures": {
          "enableNestedVirtualization": true
        }
      },
      "selfLink": "https://www.googleapis.com/compute/v1/projects/weblate-383021/global/instanceTemplates/k8s-master-20230426115237974800000002"
    }
  ],
  "selfLink": "https://www.googleapis.com/compute/v1/projects/weblate-383021/global/instanceTemplates"
}

-- body end --
total round trip time (request+response): 1.018 secs
---- response end ----
----------------------
NAME                                   MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP
k8s-master-20230426115237974800000002  n1-standard-2               2023-04-26T06:27:52.580-07:00

Some sensitive info removed from output or replaced.

edwardmedia commented 1 year ago

@Varrkan82 thanks for the log. I see where the problem is in the provider. The filter field needs translation and encoding.

For this particular case (~, equal), here is what we stand.

What it currently sends:

filter=name+~+k8s-worker-

What the api needs

filter=name+eq+%22.%2A%28k8s-worker-%29.%2A%22

karolgorc commented 1 month ago

A lot has changed in gcloud's approach to this. First of all they changed their used API method from instanceTemplates.list to instanceTemplates.aggregatedList. I also think that the filtering in gcloud isn't done using the API and rather done on the Python's side of things judging my my runs of this command:

gcloud compute instance-templates list --filter="name ~ k8s-master-"  --log-http

Also implementing something like this into the code could be tricky because of the more complex queries using RE in the filter field For example:

filter = "(name = \"tf-test-template-a\") AND (description = \"Example template.\")"

This kind of documentation is only used by 3 resources:

filter - (Optional) A filter to retrieve the instance templates. See gcloud topic filters for reference. If multiple instance templates match, either adjust the filter or specify most_recent. One of name, filter or self_link_unique must be provided.

compute_instance_template
compute_region_instance_template
compute_snapshot

I think we should change the docs to reference the filter field in the API method's documentation for these 3 resources instead of "regexing" our way into supporting the ~ operand. This is also more representative of what the code actually does because the filter in terraform is directly passed into the API request

hashicorp / terraform-provider-google