Running Packetbeat in Kubernetes

[jerroydmoore]

Hey Elastic!

I wasn't sure where a good place for this was, but I thought other users might benefit from how I setup packetbeat in k8s.

Here is my daemonset configuration. Note that using a daemonset instead of a Pod ensures that each physical host gets a packetbeat container

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: packetbeat-v5.2.1
  namespace: kube-system
  labels:
    k8s-app: packetbeat
    kubernetes.io/cluster-service: "true"
    version: v5.2.1
spec:
  template:
    metadata:
      labels:
        k8s-app: packetbeat
        kubernetes.io/cluster-service: "true"
        version: v5.2.1
    spec:
      hostNetwork: true
      containers:
      - name: packetbeat
        image: docker.elastic.co/beats/packetbeat:5.2.1
        command:
          - '/bin/sh'
          - '-c'
          - '/usr/share/packetbeat/packetbeat -e -c /etc/packetbeat/packetbeat.yml -path.logs /var/log/packetbeat'
        securityContext:
          #privileged: true
          capabilities:
            add:
              - NET_ADMIN
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlogpacketbeat
          mountPath: /var/log/packetbeat
        - name: packetbeatconf
          mountPath: /etc/packetbeat
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlogpacketbeat
        hostPath:
          path: /var/log/packetbeat
      - name: packetbeatconf
        configMap:
          name: packetbeat-conf

One of the things I've done for metricbeat is create an put my ElasticSearch endpoint in an ExternalName Service, so that in my metricbeat.yml, I can point my elasticsearch destination to this dns name rather than the actual dns name. This is the preferred pattern for routing network traffic to destinations outside your k8s cluster. This works great for metricbeat, however, breaks for packetbeat. I suspect, because packetbeat requires hostNetwork: true, it bypasses kube-dns, which holds k8s service information. The work around is to point packetbeat directly to your elasticsearch, not using k8s services. Here is my k8s service for reference if someone is able to figure out how to use k8s services on the host network, that would be fantastic.

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Elasticsearch
spec:
  type: ExternalName
  externalName: your-elasticsearch-hostname.us-west-2.es.amazonaws.com
  ports:
  - name: esrest
    port: 9200
    protocol: TCP
    targetPort: 80

Then in your metricbeat.yml/packetbeat.yml configuration

output.elasticsearch:
  hosts: ["elasticsearch-logging:"]

[exekias]

Thank you for the contribution @jerroydmoore! Kubernetes support is something we definitely want to improve, and these contributions make our work easier!

As this is not exactly an issue with current images I'm closing this thread, most likely this kind of conversation fits better in Beats discuss forum, but as I said: thank you so much, contributions and experiences are really welcomed :+1:

[manuel-palacio]

@jerroydmoore Excellent, thank you! I have pods running in different namespaces. Do you happen to know if it's possible to somehow capture the k8s namespace and send it to kibana with the rest of the data from packetbeat?

[jerroydmoore]

Hi @manuel-palacio I'm glad you found it useful! Unfortunately, the packet would have to hook into kubernetes API. Since I have limited knowledge in golang or kubernetes, (and I have higher priority work than adding this metadata), I have not looked into it further. Kubernetes does have an extensive API. The fluentd agent kubernetes ships out of the box has kubernetes metadata attached to the record before being sent to elasticsearch. Maybe you can borrow from this metadata plugin.