search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.29k stars 1.72k forks source link

VPC Flow Log filter expression doen't allow to use "psc" field #19239

Closed andy65535 closed 1 day ago

andy65535 commented 3 weeks ago

Community Note

Terraform Version & Provider Version(s)

Terraform v1.5.5 on MacOS

Affected Resource(s)

google_compute_subnetwork

Terraform Configuration

Debug Output

No response

Expected Behavior

VPC flow logs should be modified to log only flow with PSC as source or destination. 

module "vpc-test" {
  source  = "terraform-google-modules/network/google"
  version = "~> 9.1"

  project_id   = var.gcp_project_id
  network_name = "test-vpc-01"
  routing_mode = "GLOBAL"

  subnets = [
    {
      subnet_name           = "subnet-04"
      subnet_ip             = "10.1.1.0/24"
      subnet_region         = local.region
      subnet_private_access = "true"
      subnet_flow_logs          = "true"
      subnet_flow_logs_sampling = 0.75
      subnet_flow_logs_interval = "INTERVAL_1_MIN"
      subnet_flow_logs_metadata = "INCLUDE_ALL_METADATA"
+      subnet_flow_logs_filter = "has(psc.psc_endpoint.vpc.vpc_name)"
    }
  ]

  secondary_ranges = {
  }
}

Actual Behavior

Terraform will perform the following actions:

  # module.vpc-test.module.subnets.google_compute_subnetwork.subnetwork["europe-west2/subnet-04"] will be updated in-place
  ~ resource "google_compute_subnetwork" "subnetwork" {
        id                         = "projects/XXXXXXXXX/regions/europe-west2/subnetworks/subnet-04"
        name                       = "subnet-04"
        # (12 unchanged attributes hidden)

      ~ log_config {
          ~ filter_expr          = "true" -> "has(psc.psc_endpoint.vpc.vpc_name)"
            # (4 unchanged attributes hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.vpc-test.module.subnets.google_compute_subnetwork.subnetwork["europe-west2/subnet-04"]: Modifying... [id=projects/XXXXXXXXX/regions/europe-west2/subnetworks/subnet-04]
╷
│ Error: Error updating Subnetwork "projects/XXXXXXXXX/regions/europe-west2/subnetworks/subnet-04": googleapi: Error 400: Invalid value for field 'resource.logConfig.filterExpr': 'has(psc.psc_endpoint.vpc.vpc_name)'. ERROR: VPC Flow Log filter expression:1:5: undeclared reference to 'psc' (in container '')
│   | has(psc.psc_endpoint.vpc.vpc_name)
│   | ....^, invalid
│
│   with module.vpc-test.module.subnets.google_compute_subnetwork.subnetwork["europe-west2/subnet-04"],
│   on .terraform/modules/vpc-test/modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
│   28: resource "google_compute_subnetwork" "subnetwork" {
│

If I use, f.e. subnet_flow_logs_filter = "has(src_vpc.vpc_name)" it works as expected.

Steps to reproduce

  1. terraform apply

Important Factoids

I'm following the Google documentation page https://cloud.google.com/vpc/docs/about-flow-logs-records#record_format where "psc" described ad Metadata field and as far I have "INCLUDE_ALL_METADATA" I expected "psc" field will be there. Also I see "psc" field in the Logs Explorer.

References

No response

b/364879316

ggtisc commented 2 weeks ago

Hi @andy65535!

To replicate this issue we need the source code of your module terraform-google-modules/network/google and all other things that we don't have access to. If you have sensitive data you could use examples like:

project = "my-project" org_id = 1234567890 user = "my-user@my-domain.com"

andy65535 commented 1 week ago

Hi @andy65535!

To replicate this issue we need the source code of your module terraform-google-modules/network/google and all other things that we don't have access to. If you have sensitive data you could use examples like:

It is not my module. This is Google module: https://github.com/terraform-google-modules/terraform-google-network/tree/master

project = "my-project" org_id = 1234567890 user = "my-user@my-domain.com"

ggtisc commented 1 week ago

Thanks @andy65535, unfortunately we are not responsible for maintaining terraform modules. After searching I have not found the google_compute_subnetwork resource code configuration, so if you could provide the code you are using in that module, it will be helpful to reproduce this issue.

andy65535 commented 1 week ago

ok. I rewrote TF code without modules:

resource "google_compute_network" "vpc_test_network" {
  project                 = var.gcp_project_id
  name                    = "vpc-test-network"
  routing_mode            = "GLOBAL"
  auto_create_subnetworks = "false"
}

resource "google_compute_subnetwork" "network-with-private-secondary-ip-ranges" {
  name                     = "test-subnetwork"
  project                  = var.gcp_project_id
  ip_cidr_range            = "10.2.0.0/16"
  region                   = "europe-west2"
  network                  = google_compute_network.vpc_test_network.id
  private_ip_google_access = "true"

  log_config {
    aggregation_interval = "INTERVAL_10_MIN"
    flow_sampling        = 0.75
    metadata             = "INCLUDE_ALL_METADATA"
    filter_expr          = "has(psc.psc_endpoint.vpc.vpc_name)"
  }
}

apply this lead to raising the error 

google_compute_subnetwork.network-with-private-secondary-ip-ranges: Modifying... [id=projects/XXXXXXXXXX/regions/europe-west2/subnetworks/test-subnetwork]
╷
│ Error: Error updating Subnetwork "projects/XXXXXXXXXXXX/regions/europe-west2/subnetworks/test-subnetwork": googleapi: Error 400: Invalid value for field 'resource.logConfig.filterExpr': 'has(psc.psc_endpoint.vpc.vpc_name)'. ERROR: VPC Flow Log filter expression:1:5: undeclared reference to 'psc' (in container '')
│   | has(psc.psc_endpoint.vpc.vpc_name)
│   | ....^, invalid
│
│   with google_compute_subnetwork.network-with-private-secondary-ip-ranges,
│   on vpc-test.tf line 9, in resource "google_compute_subnetwork" "network-with-private-secondary-ip-ranges":
│    9: resource "google_compute_subnetwork" "network-with-private-secondary-ip-ranges" {
│
╵
ggtisc commented 1 week ago

Confirmed issue after applying the shared configuration it returns the next error:

Error: Error creating Subnetwork: googleapi: Error 400: Invalid value for field 'resource.logConfig.filterExpr': 'has(psc.psc_endpoint.vpc.vpc_name)'. ERROR: VPC Flow Log filter expression:1:5: undeclared reference to 'psc' (in container '')
│   | has(psc.psc_endpoint.vpc.vpc_name)
│   | ....^, invalid

It is not clear at all the correct nomenclature for this filter expression. There aren't examples of this in terraform registry to confirm how to manage this configuration.

andy65535 commented 1 week ago

I created the filter expression by analogy with the working expression from documentation "has(dest_vpc.vpc_name)". I attached screenshot of log from Logs Explorer where you can see these fields underlined in green. What does not work (what is this topic about) is underlined in red. And, I repeat, I rely on the description of the fields by which I can filter what is described here: https://cloud.google.com/vpc/docs/about-flow-logs-records#record_format

google filter psc

In my case we are using Cloud Composer (managed by Google) in a project and some internal communications go through PSC. What's why I have psc in logs.

andy65535 commented 1 day ago

Google fixed this in scope of Case 3154088. Now it works. This issue can be closed.

andy65535 commented 1 day ago

Close.