[Filebeat] Enhance gcp.flowlogs to support GKE flow metadata

[andrewkroh]

GCP flow logs have been updated to contain Google Kubernetes Engine (GKE) metadata. The module should be updated to handle this metadata and populate the appropriate ECS fields. The metadata contains details about the cluster, pod, and service.

• https://cloud.google.com/vpc/docs/flow-logs#gke-annotations

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[legoguy1000]

@andrewkroh does this still need to be done?

[andrewkroh]

It does, but I would only make the change to the integration package for this new feature.

https://github.com/elastic/integrations/blob/master/packages/gcp/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml

[legoguy1000]

If you have the sample data, I have no problem doing it for whichever repo you guys want.

[andrewkroh]

I enabled flow logging for a VPC that had k8s cluster. Here are some samples of the raw event.original values for one flow.

{
  "insertId": "1b6a6nrg2x5m73v",
  "jsonPayload": {
    "bytes_sent": "83",
    "connection": {
      "dest_ip": "10.4.3.4",
      "dest_port": 53,
      "protocol": 17,
      "src_ip": "10.4.0.3",
      "src_port": 54898
    },
    "dest_gke_details": {
      "cluster": {
        "cluster_location": "us-central1-a",
        "cluster_name": "analysis-cluster"
      },
      "pod": {
        "pod_name": "kube-dns-6c7b8dc9f9-jg27s",
        "pod_namespace": "kube-system"
      },
      "service": [
        {
          "service_name": "kube-dns",
          "service_namespace": "kube-system"
        }
      ]
    },
    "dest_instance": {
      "project_id": "MY_PROJECT_ID",
      "region": "us-central1",
      "vm_name": "gke-analysis-cluster-pool-1-fbe07541-c0eo",
      "zone": "us-central1-a"
    },
    "dest_vpc": {
      "project_id": "MY_PROJECT_ID",
      "subnetwork_name": "default",
      "vpc_name": "default"
    },
    "end_time": "2021-07-28T18:35:46.666932790Z",
    "packets_sent": "1",
    "reporter": "SRC",
    "src_gke_details": {
      "cluster": {
        "cluster_location": "us-central1-a",
        "cluster_name": "analysis-cluster"
      },
      "pod": {
        "pod_name": "kafka-operator-operator-67b6c6568-d6w4l",
        "pod_namespace": "kafka"
      }
    },
    "src_instance": {
      "project_id": "MY_PROJECT_ID",
      "region": "us-central1",
      "vm_name": "gke-analysis-cluster-pool-1-fbe07541-6jn3",
      "zone": "us-central1-a"
    },
    "src_vpc": {
      "project_id": "MY_PROJECT_ID",
      "subnetwork_name": "default",
      "vpc_name": "default"
    },
    "start_time": "2021-07-28T18:35:46.666932790Z"
  },
  "logName": "projects/MY_PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows",
  "receiveTimestamp": "2021-07-28T18:40:51.171122652Z",
  "resource": {
    "labels": {
      "location": "us-central1-a",
      "project_id": "MY_PROJECT_ID",
      "subnetwork_id": "5555555555555555555",
      "subnetwork_name": "default"
    },
    "type": "gce_subnetwork"
  },
  "timestamp": "2021-07-28T18:40:51.171122652Z"
}
{
  "insertId": "y4m9fmg323qpes",
  "jsonPayload": {
    "bytes_sent": "179",
    "connection": {
      "dest_ip": "10.4.0.3",
      "dest_port": 54898,
      "protocol": 17,
      "src_ip": "10.4.3.4",
      "src_port": 53
    },
    "dest_gke_details": {
      "cluster": {
        "cluster_location": "us-central1-a",
        "cluster_name": "analysis-cluster"
      },
      "pod": {
        "pod_name": "kafka-operator-operator-67b6c6568-d6w4l",
        "pod_namespace": "kafka"
      }
    },
    "dest_instance": {
      "project_id": "MY_PROJECT_ID",
      "region": "us-central1",
      "vm_name": "gke-analysis-cluster-pool-1-fbe07541-6jn3",
      "zone": "us-central1-a"
    },
    "dest_vpc": {
      "project_id": "MY_PROJECT_ID",
      "subnetwork_name": "default",
      "vpc_name": "default"
    },
    "end_time": "2021-07-28T18:35:46.668636429Z",
    "packets_sent": "1",
    "reporter": "SRC",
    "src_gke_details": {
      "cluster": {
        "cluster_location": "us-central1-a",
        "cluster_name": "analysis-cluster"
      },
      "pod": {
        "pod_name": "kube-dns-6c7b8dc9f9-jg27s",
        "pod_namespace": "kube-system"
      },
      "service": [
        {
          "service_name": "kube-dns",
          "service_namespace": "kube-system"
        }
      ]
    },
    "src_instance": {
      "project_id": "MY_PROJECT_ID",
      "region": "us-central1",
      "vm_name": "gke-analysis-cluster-pool-1-fbe07541-c0eo",
      "zone": "us-central1-a"
    },
    "src_vpc": {
      "project_id": "MY_PROJECT_ID",
      "subnetwork_name": "default",
      "vpc_name": "default"
    },
    "start_time": "2021-07-28T18:35:46.668636429Z"
  },
  "logName": "projects/MY_PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows",
  "receiveTimestamp": "2021-07-28T18:40:23.012526681Z",
  "resource": {
    "labels": {
      "location": "us-central1-a",
      "project_id": "MY_PROJECT_ID",
      "subnetwork_id": "5555555555555555555",
      "subnetwork_name": "default"
    },
    "type": "gce_subnetwork"
  },
  "timestamp": "2021-07-28T18:40:23.012526681Z"
}

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)