Closed doge95 closed 3 years ago
If we use parse type "none", then Fluentd able to tail all the logs.
## Fluentd configurations:
fileConfigs:
01_sources.conf: |-
<source>
@type tail
@log_level debug
@id in_tail_container_logs
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
exclude_path ["/var/log/containers/*_kube-system_*.log", "/var/log/containers/*_tseacloud_*.log", "/var/log/containers/*_default_*.log", "/var/log/contain
ers/*_rook-ceph_*.log", "/var/log/containers/*fluent*.log", "/var/log/containers/*_cloudsql_*.log"]
read_from_head true
follow_inodes true
<parse>
@type none
</parse>
</source>
read_bytes_limit_per_second
which is introduced at v1.13
Describe the bug
We were doing stress test on our EFK stack. We deploy a simple openresty web server that writes access logs to stdout, and use vetega tool to send HTTP requests at a constant rate (QPS: 10000 for 180 seconds). We find that when log file is rotated quite fast, it takes a long time (around 2 mins) for in_tail to discover the new log file, and in_tail loses some of the rotated files.
To Reproduce
Docker logging driver config:
Expected behavior We expect Kibana to also receive 1799999 logs. However, it only receives 622756 entires.
Your Environment
v1.12.0
Debian GNU/Linux 10 (buster)
5.4.0-48-generic
Your Configuration
Additional context Docker logs
Get the last log at 22:24. It is at line number 588614 of the log file - 8a703c5ad9672f8f8101d6ad47c4ed61b4b04d69621e84cbd0851ba1ee3d3d5f-json.log.3
Get the first log at 22:28. It is at the first line of the log file - 8a703c5ad9672f8f8101d6ad47c4ed61b4b04d69621e84cbd0851ba1ee3d3d5f-json.log
It proofs that in_tail fails to check below 2 files when log rotation happening quite fast.