Fluentd addon and Kibana Fields

[trulede]

Hi,

today I tried the Fluentd addon. It seems to work nicely, everything is more or less hooked-up correctly. All the logs coming from Pods are available. Great.

My question is relating to Kibana and the "Available Fields". Here I only really have objects listed for docker & kubernetes, and not the fields contained within those objects (such as kubernetes.pod_name). I can filter on those fields, so the data is somehow available, just not as an "Available Field".

If in Kibana I uncheck "Hide Missing Fields" then its possible to add these fields, but then they will not contain any data. So I guess that somehow they are not being parsed.

Any ideas on where to look for getting this to work? I got a feeling somewhere a bit of config is missing.

[trulede]

The fluend configuration seems OK, using Kubernetes fluentd plugin, the JSON seems to be formed as expected.

{
  "_index": "logstash-2020.03.20",
  "_type": "fluentd",
  "_id": "jl8c-HABNWcTuaaewzvY",
  "_version": 1,
  "_score": null,
  "_source": {
    "stream": "stderr",
    "log": "ERROR: logging before flag.Parse: E0320 13:23:14.305157       1 nanny_lib.go:110] the server could not find the requested resource",
    "docker": {
      "container_id": "1267e4a88b1a9f34ae9d73a0d77a55ea88e19ad31f40fdbbccf7a2c1bcaab4f1"
    },
    "kubernetes": {
      "container_name": "eventer-nanny",
      "namespace_name": "kube-system",
      "pod_name": "heapster-v1.5.2-5c58f64f8b-9zkpf",
      "container_image": "docker.io/cdkbot/addon-resizer-amd64:1.8.1",
      "container_image_id": "docker.io/cdkbot/addon-resizer-amd64@sha256:a5ff31fb60d32e02780441fc81bc91dd549097d6afeef3c6decb6005289795af",
      "pod_id": "dc321a8e-32a7-4381-b13c-8bc8000b79c2",
      "labels": {
        "k8s-app": "heapster",
        "pod-template-hash": "5c58f64f8b",
        "version": "v1.5.2"
      },
      "host": "ubuntu-bionic",
      "master_url": "https://10.152.183.1:443/api",
      "namespace_id": "defcdb10-74b8-41ac-a2f3-ab437683daad"
    },
    "@timestamp": "2020-03-20T13:23:14.305232491+00:00",
    "tag": "kubernetes.var.log.containers.heapster-v1.5.2-5c58f64f8b-9zkpf_kube-system_eventer-nanny-1267e4a88b1a9f34ae9d73a0d77a55ea88e19ad31f40fdbbccf7a2c1bcaab4f1.log"
  },
  "fields": {
    "@timestamp": [
      "2020-03-20T13:23:14.305Z"
    ]
  },
  "sort": [
    1584710594305
  ]
}

All items directly under "_source" are parsed as Available Fields, but not their children items. So the list in the Kibana giu includes _id, _index, _type, docker, kubernetes, log, stream and tag. The items under kubernetes and docker are not listed as "Available Fields".

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.