Closed andrewkroh closed 2 years ago
Filebeat's spec currently filters by input type, but it happens too late. If that filter is moved earlier it should fix the issue. Ideally inputs types destined for Filebeat would follow naming convention like the other Beats (Metricbeat uses .*/metric
, Auditbeat uses audit/.*
, Packetbeat uses type: packet
, etc.).
diff --git a/internal/spec/filebeat.yml b/internal/spec/filebeat.yml
index 10f8ee449..626d6e241 100644
--- a/internal/spec/filebeat.yml
+++ b/internal/spec/filebeat.yml
@@ -12,6 +12,42 @@ artifact: beats/filebeat
restart_on_output_change: true
rules:
- fix_stream: {}
+
+- filter_values:
+ selector: inputs
+ key: type
+ values:
+ - aws-cloudwatch
+ - aws-s3
+ - azure-eventhub
+ - cloudfoundry
+ - container
+ - docker
+ - event/file
+ - event/stdin
+ - event/tcp
+ - event/udp
+ - filestream
+ - gcp-pubsub
+ - http_endpoint
+ - httpjson
+ - journald
+ - kafka
+ - log/docker
+ - log/redis_slowlog
+ - log/syslog
+ - logfile
+ - mqtt
+ - netflow
+ - o365audit
+ - redis
+ - stdin
+ - syslog
+ - tcp
+ - udp
+ - unix
+ - winlog
+
- inject_index:
type: logs
@@ -63,34 +99,6 @@ rules:
- remove_key:
key: data_stream.dataset
-- filter_values:
- selector: inputs
- key: type
- values:
- - aws-cloudwatch
- - aws-s3
- - azure-eventhub
- - cloudfoundry
- - container
- - docker
- - gcp-pubsub
- - http_endpoint
- - httpjson
- - journald
- - kafka
- - log
- - mqtt
- - netflow
- - o365audit
- - redis
- - stdin
- - syslog
- - tcp
- - udp
- - unix
- - winlog
- - filestream
-
- filter_values:
selector: inputs
key: enabled
@cmacknz it rather seems to be a data plane issue, am I wrong?
I don't know that anyone on the data plane team has touched the spec files before, but the plan for V2 is that we would own this so I think we can own the fix for this.
Pulling into 8.4 as we are getting multiple reports about this problem now.
@andrewkroh do you know which versions this is affecting? All of them? Or just recent 8.x releases? I'm wondering what backports we need.
We should likely also link this problem and the work around in the agent release notes.
My assumption is that 7.17 and 8.x are affected because those are the stack versions supported by network_traffic (https://github.com/elastic/integrations/blob/21cf4a1e6d80e01ee29651f58ba5cb7455cdad0c/packages/network_traffic/manifest.yml#L12).
I was on 8.2 where it was working for me.
When an agent policy contains the Packetbeat redis input, Elastic Agent is generating config for Filebeat that includes a redis log input. The policy for Packetbeat uses input
type: packet
with a data_stream oftype: redis
. For example:The impact is that this causes Filebeat to report UNHEALTHY status if Packetbeat is deployed at the same time. A workaround is to disable redis collection in the Network Packet Capture integration.
Here is a patch to the Elastic Agent testdata that reproduces the bug using unit tests:
Workarounds
You can disable the redis protocol in the network packet capture integration.