[ECK Filebeat] Harvesting with autodiscover works in 8.5.3 but not with >8.6.0

[tmsdce]

Hi,

Filebeat is deployed using the Beats CRD provided by the ECK operator. The following configuration works with version 8.5.3 but not with newer versions (>8.6.0)

filebeat:
  autodiscover:
    providers:
      - hints:
        default_config:
          enabled: false
        node: ${NODE_NAME}
        templates:
        - condition:
          or:
            - equals:
              kubernetes:
                labels:
                  my_custom_label/key: myvalue
            - equals:
              kubernetes:
                labels:
                  my_other_custom_label/key: myothervalue
          config:
          - paths:
            - /var/log/containers/*${data.kubernetes.container.id}.log
            type: container
        type: kubernetes
output:
  logstash:
    hosts:
      - myhost:5044
    ssl:
      certificate: /etc/filebeat/tls/logstash/tls.crt
      certificate_authorities:
        - /etc/filebeat/tls/logstash/ca.crt
      key: /etc/filebeat/tls/logstash/tls.key
processors:
  - add_fields:
    fields:
      host:
        name: ${NODE_NAME}
    target: ""

• When using version 8.5.3, filebeat logs shows harvester started for various containers logs matched by the specified labels in template conditions:

  {"log.level":"info","@timestamp":"2024-06-19T15:59:06.441Z","log.logger":"input.harvester","log.origin":{"file.name":"log/harvester.go","file.line":310},"message":"Harvester started for paths: [/var/log/containers/*afec2ad8cb5b6ceb0b25bf4f83cb20e8857981055b58f0c9eb257fea65eb5dc3.log]","service.name":"filebeat","input_id":"e1521747-6813-4bb5-bcaf-0788f4518a9a","source_file":"/var/log/containers/mycontainer-afec2ad8cb5b6ceb0b25bf4f83cb20e8857981055b58f0c9eb257fea65eb5dc3.log","state_id":"native::131125-2080","finished":false,"os_id":"131125-2080","harvester_id":"f2e3bbae-8093-4112-ab17-1d586df03b3e","ecs.version":"1.6.0"}

• When using version 8.6.0 or newer (I also tried with the latest version 8.14.1), no harvester ever starts. The only thing I get is this kind of logs, showing no files detected for harvesting:

  {"log.level":"info","@timestamp":"2024-06-19T16:15:16.684Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":36306944}}}},"cpu":{"system":{"ticks":30},"total":{"ticks":160,"value":160},"user":{"ticks":130}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"8f0245c4-d538-4767-ad8d-435ae999318a","uptime":{"ms":60054},"version":"8.6.0"},"memstats":{"gc_next":20095496,"memory_alloc":12040824,"memory_total":56438680,"rss":110301184},"runtime":{"goroutines":54}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.06,"15":0.1,"5":0.1,"norm":{"1":0.01,"15":0.0167,"5":0.0167}}}},"ecs.version":"1.6.0"}}

It's as if the configuration was not loaded at all with >8.6.0.

Maybe I'm missing a configuration for versions >8.6.0 but I can't find which one. In the release notes between 8.5.3 and 8.6.0, I don't see a relevant change.

[jiubujian]

I also encountered the same issue. I originally followed the latest documentation use Filebeat in Docker, but no matter what I tried, it didn't work. Until I saw this issue, I downgraded Filebeat from 8.14.1 to 8.5.3, and then the same configuration file worked. Furthermore, from version 7.0 onwards, the module "apache2" has been renamed to "apache", but the latest documentation still uses "apache2".

[jiubujian]

I have tested all the 8.x.0 images starting from 8.5.3, and found that from 8.12.0 onwards, the autodiscover feature no longer works properly, even though I strictly followed the documentation and only modified the image tag version. If I was wrong, please let me know.

[cmacknz]

Is this specific to Filebeat on ECK? Or is Filebeat running separate from ECK?

[tmsdce]

@cmacknz In my case, I only deploy my filebeat agent through ECK using the Beat CRD. ECK Operator's version is 2.11.1

For reference, here's the complete Beat spec I use.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: mybeat
  namespace: myns
spec:
  type: filebeat
  version: 8.5.3
  config:
    filebeat.autodiscover.providers:
      - hints.default_config.enabled: false
        node: '${NODE_NAME}'
        templates:
          - condition:
              or:
                - equals:
                    my_custom_label/key: myvalue
                - equals:
                    my_other_custom_label/key: myothervalue
            config:
              - paths:
                  - '/var/log/containers/*${data.kubernetes.container.id}.log'
                type: container
        type: kubernetes
    output:
      logstash:
        hosts:
          - myhost:5044
        ssl:
          certificate: /etc/filebeat/tls/logstash/tls.crt
          certificate_authorities:
            - /etc/filebeat/tls/logstash/ca.crt
          key: /etc/filebeat/tls/logstash/tls.key
    processors:
      - add_fields:
          fields:
            host.name: '${NODE_NAME}'
          target: ''
  daemonSet:
    podTemplate:
      spec:
        automountServiceAccountToken: true
        containers:
          - env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
            name: filebeat
            resources:
              limits:
                memory: 256Mi
              requests:
                cpu: 50m
                memory: 256Mi
            securityContext:
              runAsUser: 0
            volumeMounts:
              - mountPath: /etc/filebeat/tls/logstash
                name: custom-tls
                readOnly: true
              - mountPath: /var/log/containers
                name: varlogcontainers
                readOnly: true
              - mountPath: /var/log/pods
                name: varlogpods
                readOnly: true
              - mountPath: /var/lib/docker/containers
                name: varlibdockercontainers
                readOnly: true
        serviceAccountName: mysa
        terminationGracePeriodSeconds: 30
        tolerations:
          - operator: Exists
        volumes:
          - name: custom-tls
            secret:
              secretName: logstash-tls
          - hostPath:
              path: /var/log/containers
            name: varlogcontainers
          - hostPath:
              path: /var/log/pods
            name: varlogpods
          - hostPath:
              path: /var/lib/docker/containers
            name: varlibdockercontainers

[jiubujian]

@cmacknz I deployed a brand new test environment using Docker Compose, and as long as I changed the Filebeat version from 8.14.1 to 8.11.0, I was able to see the logs in Kibana. Here is an example of the configuration file:

# docker-compose.yml
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    volumes:
      - ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    user: root
    command: ["-e", "-strict.perms=false"]

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.14.1
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - 9200:9200

  kibana:
    image: docker.elastic.co/kibana/kibana:8.14.1
    ports:
      - 5601:5601

# filebeat.docker.yml
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

[tmsdce]

Hi @cmacknz Do you have any updates on this issue ?