CI fails when updating to 8.12.1

[mgsotelo]

Problem description

It seems that something is preventing filebeat to connect with elasticsearch and sending docs to it when using v8.12.1, and that is stopping the normal flow of Pull Requests using the bot.

Extra information

For testing purposes, I've added the following to the run-tests-filebeat.sh script.

# retry for max 90s (30*3s) 
# increased timeout
for _ in $(seq 1 30); do
    response_all="$(curl "http://${ip_es}:9200/filebeat-*/_search?q=agent.type:%22filebeat%22" -s -u elastic:testpasswd)" 
    # I want at least something from agent.type = "filebeat" (it fails)
    # more code, unchanged
done

Stack configuration

Docker setup

$ docker version

Github's docker version 
(I'm only trying to use the bot to update my repo, and it seems this action in your repo fails too)

$ docker-compose version

Github's docker compose version 
(I'm only trying to use the bot to update my repo, and it seems this action in your repo fails too)

Container logs

$ docker-compose logs

# Can't provide it, only the output from the pipeline...

[+] Searching all documents generated by Filebeat

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
0
{"took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0.0,"hits":[]}}
Expected at least 1 document
Error: Process completed with exit code 1.

[antoineco]

Indeed:

• 954

• 953

I didn't have time to investigate unfortunately.

[antoineco]

The issue seems to be related to Autodiscover.

The docker provider — which we have enabled by default — is supposed to detect Docker containers, and launch the proper config for them:

https://github.com/deviantony/docker-elk/blob/59fcf834b285365cbdca56fc76b49fc9abe99a7f/extensions/filebeat/config/filebeat.yml#L12-L17

Since 8.12, the generated config seems incorrect. For instance, I see the following warning about the template trying to resolve the field data.kubernetes.container.id, whereas I would expect that field to be data.docker.container.id:

{
  "log.level": "warn",
  "@timestamp": "2024-02-12T08:23:17.875Z",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate",
    "file.name": "template/config.go",
    "file.line": 157
  },
  "message": "autodiscover: Configuration template cannot be resolved: field data.kubernetes.container.id not available in event or environment accessing id",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

If I declare a container input manually, container logs are being sent to Elasticsearch.

--- a/extensions/filebeat/config/filebeat.yml
+++ b/extensions/filebeat/config/filebeat.yml
@@ -16,6 +16,11 @@ filebeat.autodiscover:
     - type: docker
       hints.enabled: true

+filebeat.inputs:
+- type: container
+  paths:
+    - '/var/lib/docker/containers/*/*.log'
+

Let me troubleshoot this further when I find the time.

[antoineco]

Culprit found: https://github.com/elastic/beats/commit/41ab08cd6aa9e2c367b0c4b670f05f786a2862db

Elastic changed the default config:

-           "/var/lib/docker/containers/${data.container.id}/*-json.log",
+           "/var/log/containers/*-${data.kubernetes.container.id}.log",

Pushing a fix right now.

[antoineco]

Fixed by 44822f6045986b589da80476214ce55e86b05677

edit: apologies for the nonsense in the commit message:

- changes the default output from 'container' to 'input'.
+ changes the default input from 'container' to 'filestream'.