Open cloud-thought opened 5 months ago
Does it happen with 3.0?
Hey @patrick-stephens, have not tried version 3.0. We are using this Helm Chart https://artifacthub.io/packages/helm/fluent/fluent-bit
I would try in case it's already resolved, in which case it's a case of figuring out what to backport or if it still affects it. I can't easily test ARM clusters I'm afraid so relies on user info.
Alright will give it a try. Nonetheless, maybe other users have similar issues and would like to report. I will post an update as soon as I have enough data.
Ok, didn't have to wait for that long to encounter the same issue:
Defaulted container "fluent-bit" out of: fluent-bit, reloader
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
___________.__ __ __________.__ __ ________
\_ _____/| | __ __ ____ _____/ |_ \______ \__|/ |_ ___ _\_____ \
| __) | | | | \_/ __ \ / \ __\ | | _/ \ __\ \ \/ / _(__ <
| \ | |_| | /\ ___/| | \ | | | \ || | \ / / \
\___ / |____/____/ \___ >___| /__| |______ /__||__| \_/ /______ /
\/ \/ \/ \/ \/
[2024/03/25 15:06:58] [engine] caught signal (SIGSEGV)
#0 0xaaaae6aec54b in cb_engine_sched_timer() at src/flb_engine.c:175
#1 0xaaaae6af09a7 in flb_sched_event_handler() at src/flb_scheduler.c:428
#2 0xaaaae6aed42f in flb_engine_start() at src/flb_engine.c:1025
#3 0xaaaae6acd9ff in flb_lib_worker() at src/flb_lib.c:674
#4 0xffffaf669647 in ???() at ???:0
#5 0xffffaefce01b in ???() at ???:0
#6 0xffffffffffffffff in ???() at ???:0
This time it happened on nearly all ARM64 nodes we have running in our cluster.
We are hitting this too, and it still occurs with 3.0.7. Multiple clusters on EKS (v1.28.9) on arm64.
The problem seems related to the kubernetes_events
input, because our clusters have two separate deployments of Fluent Bit: one deployed as a DaemonSet to collect pod logs, and the other as a Deployment to collect K8s events. The pod logs instances are fine, it's only the events instances that are crashing. And the key difference is that the events deployment uses the kubernetes_events
input while the pods deployment doesn't.
Here's the config we are using (except obviously the endpoint FQDN has been redacted):
service:
daemon: false
health_check: true
hot_reload: true
http_listen: 0.0.0.0
http_port: 2020
http_server: true
parsers_file: /fluent-bit/etc/conf/parsers.conf
scheduler.cap: 300
pipeline:
inputs:
- alias: kubernetes_events
db: /data/events.db
name: kubernetes_events
tag: kube.*
outputs:
- alias: loki
compress: gzip
drop_single_key: true
host: loki.example.com
labels: component=kubernetes, environment=lab, service=somesvc, cluster=some-eks-cluster, namespace=$metadata['namespace'], region=us-east-1, source=events
match: '*'
name: loki
port: 443
retry_limit: 500
tls: true
tls.crt_file: /secrets/cert/tls.crt
tls.key_file: /secrets/cert/tls.key
tls.verify: true
Log output:
Fluent Bit v3.0.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
___________.__ __ __________.__ __ ________
\_ _____/| | __ __ ____ _____/ |_ \______ \__|/ |_ ___ _\_____ \
| __) | | | | \_/ __ \ / \ __\ | | _/ \ __\ \ \/ / _(__ <
| \ | |_| | /\ ___/| | \ | | | \ || | \ / / \
\___ / |____/____/ \___ >___| /__| |______ /__||__| \_/ /______ /
\/ \/ \/ \/ \/
[2024/06/12 17:24:26] [ info] [fluent bit] version=3.0.7, commit=73e72bdf9a, pid=1
[2024/06/12 17:24:26] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/06/12 17:24:26] [ info] [cmetrics] version=0.9.1
[2024/06/12 17:24:26] [ info] [ctraces ] version=0.5.1
[2024/06/12 17:24:26] [ info] [input:kubernetes_events:kubernetes_events] initializing
[2024/06/12 17:24:26] [ info] [input:kubernetes_events:kubernetes_events] storage_strategy='memory' (memory only)
[2024/06/12 17:24:26] [ info] [input:kubernetes_events:kubernetes_events] API server: https://kubernetes.default.svc:443
[2024/06/12 17:24:26] [ info] [input:kubernetes_events:kubernetes_events] thread instance initialized
[2024/06/12 17:24:26] [ info] [output:loki:loki] configured, hostname=loki.example.com:443
[2024/06/12 17:24:26] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/06/12 17:24:26] [ info] [sp] stream processor started
[2024/06/12 17:33:24] [engine] caught signal (SIGSEGV)
#0 0xaaaac0c7d65b in cb_engine_sched_timer() at src/flb_engine.c:175
#1 0xaaaac0c81adb in flb_sched_event_handler() at src/flb_scheduler.c:428
#2 0xaaaac0c7e58b in flb_engine_start() at src/flb_engine.c:1025
#3 0xaaaac0c5deb3 in flb_lib_worker() at src/flb_lib.c:674
#4 0xffff8d89ee57 in ???() at ???:0
#5 0xffff8d907f9b in ???() at ???:0
#6 0xffffffffffffffff in ???() at ???:0
If you remove the events input does it resolve?
I guess that question is for you @cloud-thought since in my case if I remove the events input I've got no inputs :)
Great that this is getting more attention! Well, we are not able to test it at this point in time. But as soon as we get the chance to do so, I will come back with more insights about the result of deactivating the k8s events collector.
@patrick-stephens So back with some test results. We deactivated the K8S events input and indeed, no more restarts since 1 week. Any idea what actually correlates with this behaviour?
@patrick-stephens So back with some test results. We deactivated the K8S events input and indeed, no more restarts since 1 week one. Any idea what actually correlates with this behaviour?
I don't have any knowledge of the code, I'm just good at breaking things (!) so trying to find the minimal breakage required helps narrow it down. It seems like there is something related to the events plugin then (or one of the others alongside it) so may tag in @pwhelan to see if he has some ideas.
Can you clarify the exact minimal set of plugins that both fail and do not fail for you?
Respective to the configuration in my first post, the only thing that I changed is the K8S events input.
Bug Report
Describe the bug FluentBit restarts sporadically with "[engine] caught signal (SIGSEGV)" on ARM64 Nodes in Kubernetes Cluster (AWS EKS)
To Reproduce
Not sure how to reproduce the issue. Only happens on ARM64 nodes. The FluentBit pods have enough memory assigned to them and the node is not under any type of heavy load. Our FluentBit pods on the AMD64 nodes work fine and the restarts only happen very rarely in regards on a minimal subset of our ARM64 nodes.
Expected behavior No restarts happening.
Screenshots None.
Your Environment
Tail generic container logs
[INPUT] Name tail Path /var/log/containers/.log Exclude_Path /var/log/containers/daprd.log,/var/log/containers/fluent-bit.log,/var/log/containers/fluentd.log DB /var/log/flb_containers_log.db multiline.parser cri Read_From_Head On Tag kube.generic. Refresh_Interval 10 storage.type filesystem
Tail daprd container logs
[INPUT] Name tail Path /var/log/containers/daprd.log Exclude_Path /var/log/containers/fluent-bit.log,/var/log/containers/fluentd.log DB /var/log/flb_daprd_log.db multiline.parser cri Read_From_Head On Tag kube.daprd.* Refresh_Interval 10 storage.type filesystem
Tail system logs
[INPUT] Name systemd Tag host.systemd.* DB /var/log/flb_systemd_log.db Read_From_Tail On Systemd_Filter _SYSTEMD_UNIT=kubelet.service Systemd_Filter _SYSTEMD_UNIT=kubeproxy.service Systemd_Filter _SYSTEMD_UNIT=containerd.service Strip_Underscores On
Collect Kubernetes events
[INPUT] name kubernetes_events tag host.events.*
Gather kubernetes metadata by using the Kubelet API for generic containers
[FILTER] Name kubernetes Match kube.generic.* Merge_Log On Keep_Log Off K8S-Logging.Parser On K8S-Logging.Exclude On Kube_Tag_Prefix kube.generic.var.log.containers. Use_Kubelet On Buffer_Size 0
Gather kubernetes metadata by using the Kubelet API for daprd containers
[FILTER] Name kubernetes Match kube.daprd.* Merge_Log On Keep_Log Off K8S-Logging.Parser On K8S-Logging.Exclude On Kube_Tag_Prefix kube.daprd.var.log.containers. Use_Kubelet On Buffer_Size 0
Nest metadata fields
[FILTER] Name nest Match kube.daprd.* Operation nest Wildcard app_id Wildcard scope Wildcard type Wildcard ver Nest_under dapr
Record normalization for further processing
[FILTER] Name modify Match kube.daprd.* Rename time timestamp Rename msg message
Only output logs for containers with set fluentd.active=true or fluentbit.io/active=true annotations
[FILTER] Name grep Match kube.* Logical_Op or Regex $kubernetes['annotations']['fluentd.active'] true Regex $kubernetes['annotations']['fluentbit.io/active'] true
Rewrite tag by provided custom tag for Kubernetes scaffold logs
[FILTER] Name rewrite_tag Match kube.* Rule $kubernetes['annotations']['fluentbit.io/tag'] ^scaffold$ custom.scaffold.$TAG false Emitter_Name kubernetes_scaffold_emitter
Add AWS metadata to container logs
[FILTER] Name aws Match kube.* imds_version v1 account_id true
Add AWS metadata to custom emitter logs
[FILTER] Name aws Match custom.* imds_version v1 account_id true
Remove unnecessary keys from container logs
[FILTER] Name lua Match kube.* script /fluent-bit/scripts/remove_keys.lua call remove_keys
Final record mapping for container logs
[FILTER] Name modify Match kube.* Add @log_group /eks/${CLUSTER_NAME}/containers Rename x_request_id x-request-id
Discard Kubernetes events of type normal
[FILTER] Name grep Match host.events.* Exclude type Normal
Kubernetes events final mapping
[FILTER] Name record_modifier Match host.events.* Record inputSource kubernetes_events
Remove unnecessary keys from container logs
[FILTER] Name record_modifier Match host.systemd.* Whitelist_key MESSAGE Whitelist_key HOSTNAME Whitelist_key SYSTEMD_UNIT Record inputSource systemd
Output container logs to CloudWatch grouped by EKS cluster name
[OUTPUT] Name cloudwatch_logs Match kube.* region ${REGION} log_group_name /eks/fluentbit/${CLUSTER_NAME}/containers log_stream_template $kubernetes['pod_name'].$kubernetes['namespace_name'].$kubernetes['container_name'].$kubernetes['pod_id'] log_stream_prefix fallback-stream auto_create_group On retry_limit 5 net.keepalive On net.keepalive_idle_timeout 4s log_retention_days 5
Output container logs to Elastic grouped by EKS cluster name
[OUTPUT] Name es Match kube.* Index eks-${CLUSTER_NAME}-containers cloud_id ${ELASTICSEARCH_CLOUD_ID} cloud_auth ${ELASTICSEARCH_USER}:${ELASTICSEARCH_PASSWORD} Trace_Error On tls On tls.verify Off Retry_Limit 5 write_operation upsert Replace_Dots On Generate_ID On Suppress_Type_Name On
Output Kubernetes events to CloudWatch grouped by EKS cluster name
[OUTPUT] Name cloudwatch_logs Match host.events.* region ${REGION} log_group_name /eks/fluentbit/${CLUSTER_NAME}/host log_stream_name events auto_create_group On retry_limit 5 net.keepalive On net.keepalive_idle_timeout 4s log_retention_days 5
Output host logs to CloudWatch grouped by EKS cluster name
[OUTPUT] Name cloudwatch_logs Match host.systemd.* region ${REGION} log_group_name /eks/fluentbit/${CLUSTER_NAME}/host log_stream_prefix systemd auto_create_group On retry_limit 5 net.keepalive On net.keepalive_idle_timeout 4s log_retention_days 5
Output systemd logs to Elastic grouped by EKS cluster name
[OUTPUT] Name es Match host.* Index kubernetes-host-eks-${CLUSTER_NAME} cloud_id ${ELASTICSEARCH_CLOUD_ID} cloud_auth ${ELASTICSEARCH_USER}:${ELASTICSEARCH_PASSWORD} Trace_Error On tls On tls.verify Off Retry_Limit 5 write_operation upsert Replace_Dots On Generate_ID On Suppress_Type_Name On
Output DevOps container logs to Elastic grouped by EKS cluster name
[OUTPUT] Name es Match custom.scaffold.* Index kubernetes-scaffold-eks-${CLUSTER_NAME} cloud_id ${ELASTICSEARCH_CLOUD_ID} cloud_auth ${ELASTICSEARCH_USER}:${ELASTICSEARCH_PASSWORD} Trace_Error On tls On tls.verify Off Retry_Limit 5 write_operation upsert Replace_Dots On Generate_ID On Suppress_Type_Name On