Closed mgsotelo closed 7 months ago
Indeed:
I didn't have time to investigate unfortunately.
The issue seems to be related to Autodiscover.
The docker
provider — which we have enabled by default — is supposed to detect Docker containers, and launch the proper config for them:
Since 8.12, the generated config seems incorrect. For instance, I see the following warning about the template trying to resolve the field data.kubernetes.container.id
, whereas I would expect that field to be data.docker.container.id
:
{
"log.level": "warn",
"@timestamp": "2024-02-12T08:23:17.875Z",
"log.origin": {
"function": "github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate",
"file.name": "template/config.go",
"file.line": 157
},
"message": "autodiscover: Configuration template cannot be resolved: field data.kubernetes.container.id not available in event or environment accessing id",
"service.name": "filebeat",
"ecs.version": "1.6.0"
}
If I declare a container
input manually, container logs are being sent to Elasticsearch.
--- a/extensions/filebeat/config/filebeat.yml
+++ b/extensions/filebeat/config/filebeat.yml
@@ -16,6 +16,11 @@ filebeat.autodiscover:
- type: docker
hints.enabled: true
+filebeat.inputs:
+- type: container
+ paths:
+ - '/var/lib/docker/containers/*/*.log'
+
Let me troubleshoot this further when I find the time.
Culprit found: https://github.com/elastic/beats/commit/41ab08cd6aa9e2c367b0c4b670f05f786a2862db
Elastic changed the default config:
- "/var/lib/docker/containers/${data.container.id}/*-json.log",
+ "/var/log/containers/*-${data.kubernetes.container.id}.log",
Pushing a fix right now.
Fixed by 44822f6045986b589da80476214ce55e86b05677
edit: apologies for the nonsense in the commit message:
- changes the default output from 'container' to 'input'.
+ changes the default input from 'container' to 'filestream'.
Problem description
It seems that something is preventing filebeat to connect with elasticsearch and sending docs to it when using v8.12.1, and that is stopping the normal flow of Pull Requests using the bot.
Extra information
For testing purposes, I've added the following to the
run-tests-filebeat.sh
script.Stack configuration
Docker setup
Container logs