[Question] stackdriver k8s_pod resource configuration?

[PDVJAM]

Hey! We're trying to configure stackdriver monitoring with fluent-bit on k8s. I see fresh patch that added ability to use new resource types: https://github.com/fluent/fluent-bit/pull/2281 But its unclear how to configure it? Could someone help?

My config is:

 fluent-bit.conf: |
    [SERVICE]
      Flush         1
      Log_Level     warn
      Daemon        off
      Parsers_File  parsers.conf

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-stackdriver.conf

  input-kubernetes.conf: |
    [INPUT]
      Name               tail
      Tag                k8s_pod.*
      Path               /var/log/containers/*.log
      Parser             docker

  filter-kubernetes.conf: |
    [FILTER]
      Name                kubernetes
      Match               *
      Kube_URL            https://kubernetes.default.svc.cluster.local:443
      Annotations         Off

    [FILTER]
      Name record_modifier
      Match k8s_pod.*
      Record logging.googleapis.com/local_resource_id k8s_pod.<namespace_name>.<pod_name>

  output-stackdriver.conf: |
    [OUTPUT]
      Name                       stackdriver
      Match                      k8s_pod.*
      Resource                   k8s_pod
      k8s_cluster_name           cluster-0001
      k8s_cluster_location       digitalocean
      google_service_credentials /var/secrets/google/key.json
      tls                        On
      tls.verify                 Off
      severity_key               level

  parsers.conf: |
    [PARSER]
      # https://docs.fluentbit.io/manual/parser/json
      Name        docker
      Format      json
      Time_Key    time
      Time_Format %Y-%m-%dT%H:%M:%S.%L
      Time_Keep   Off
      # Command      |  Decoder | Field | Optional Action
      # =============|==================|=================
      Decode_Field_As   json       log   do_next
      Decode_Field_As   escaped    log

And it even works, but in stackdriver log recods look like

labels: {
location: "digitalocean"
namespace_name: "<namespace_name>"
cluster_name: "cluster-0001"
project_id: "certain-router-280716"
pod_name: "<pod_name>"
}

But how can I got actual namespace name and pod names from k8s instead of and ?

We've tried something like

   [FILTER]
      Name record_modifier
      Match k8s_pod.*

      Record logging.googleapis.com/local_resource_id k8s_pod.${namespace_name}.${pod_name}

but logs show "fail to process local_resource_id from log entry for k8s_pod".

[igorpeshansky]

The pod and container information is stored as part of the tag. When fluentd is used to send Kubernetes logs to Stackdriver (Google Cloud Logging), it extracts those components of the tag and assemblies a local_resource_id value that the output plugin then turns into a monitored resource. Sadly, it looks like fluent-bit lacks the functionality to parse the tag and capture its elements into record fields (the opposite of that rewrite_tag does), so what you're trying to do is not possible today. @edsiper any plans to extend record_modifier to support capture and tag processing?

[edsiper]

no plans "yet" to support record accessor mode in that filter.

A workaround would be to add the proper environment variables to your Pod, and then let Fluent Bit use them in the configuration for record_modifier, e.g:

apiVersion: v1
kind: Pod
metadata:
  name: env-test
spec:
  containers:
    - name: env-test
      image: k8s.gcr.io/busybox
      command: [ "sh", "-c"]
      args:
      - while true; do
          echo -en '\n';
          printenv K8S_NODE_NAME K8S_POD_NAME K8S_POD_NAMESPACE;
          sleep 10;
        done;
      env:
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: K8S_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: K8S_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
  restartPolicy: Never

if you get into the pod:

$ kubectl exec --stdin --tty env-test /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ # env |grep K8S_
K8S_NODE_NAME=minikube
K8S_POD_NAME=env-test
K8S_POD_NAMESPACE=default

[edsiper]

ah, my mistake, you want the pod_name and namespace for the monitored pod, not the running Pod. that feature is not yet available.

[PDVJAM]

Yes, the idea was to have info from monitored pods or containers. No workarounds at all? It seems that stackdriver support is really on the low side yet:(

[PDVJAM]

It's been fixed here.