Open hunkeelin opened 4 years ago
Fluentd not sendings logs to es or in s3 successfully. I got a similar issue to https://github.com/fluent-plugins-nursery/fluent-plugin-concat/issues/37 ...
<match fluent.**> @type null </match> <source> @type tail path /var/log/containers/*.log pos_file /var/log/es-containers.log.pos time_format %Y-%m-%dT%H:%M:%S.%NZ tag kubernetes.* format json read_from_head false </source> <filter kubernetes.**> @type concat key log multiline_start_regexp /(?<time>\d+(?:-\d+){2}\s+\d+(?::\d+){2}\.\d+)/ flush_interval 10 </filter> <filter kubernetes.**> @type kubernetes_metadata </filter> <match kubernetes.var.log.containers.**_kube-system_**> @type null </match> <match kubernetes.var.log.containers.**_test_**> @type null </match> <match kubernetes.var.log.containers.**_monitoring_**> @type null </match> <match kubernetes.var.log.containers.**_istio-system_**> @type null </match> <match kubernetes.var.log.containers.**_default_**> @type null </match> <match kubernetes.var.log.containers.**_fb-wilson_**> @type null </match> <match kubernetes.var.log.containers.**_infrastructure_**> @type null </match> <match kubernetes.var.log.containers.**_kube-public_**> @type null </match> <match kubernetes.var.log.containers.**_kube-node-lease_**> @type null </match> <match kubernetes.var.log.containers.**_istio-proxy_**> @type null </match> <match kubernetes.var.log.containers.**_fluentd-**> @type null </match> <match kubernetes.var.log.containers.**istio-system_**> @type null </match> <match **> @type copy deep_copy true <store> @type "aws-elasticsearch-service" @log_level info include_tag_key true port 443 logstash_format true buffer_chunk_limit 5M flush_interval 5s max_retry_wait 30 disable_retry_limit num_threads 15 reload_connections false <endpoint> </endpoint> </store> <store> @type s3 buffer_path /var/log/td-agent/s3 time_slice_format %Y%m%d%H time_slice_wait 10m timekey_wait 10m flush_interval 60s buffer_chunk_limit 2M chunk_limit_size 256m buffer_queue_limit 250 utc format_json true include_time_key true include_tag_key true buffer_chunk_limit 256m </store> </match>
2020-02-29 00:06:26 +0000 [info]: #0 Timeout flush: kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log:default 2020-02-29 00:06:36 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush: kubernetes.var.log.containers.fluentd-elasticsearch-4b5sb_logging_fluentd-elasticsearch-1886c1f72c4df4b2bfa247895d81141bd4b61e6b1a224f97ca836750fce7114b.log:default" location=nil tag="kubernetes.var.log.containers.fluentd-elasticsearch-4b5sb_logging_fluentd-elasticsearch-1886c1f72c4df4b2bfa247895d81141bd4b61e6b1a224f97ca836750fce7114b.log" time=#<Fluent::EventTime:0x007f3e94211b08 @sec=1582934796, @nsec=337412517> record={"log"=>"2020-02-29 00:06:26 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ConcatFilter::TimeoutError error=\"Timeout flush: kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log:default\" location=nil tag=\"kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log\" time=#<Fluent::EventTime:0x007f3e91789890 @sec=1582934786, @nsec=336591187> record={\"log\"=>\"2020-02-29 00:06:16.065 INFO 1 --- [ Thread-3] b.c.c.NetworkCardTransactionEventHandler : Ignoring ISO8583 network management request with eventId:568e6a6d-5e00-4a30-be26-5d304c83357b\\n\", \"stream\"=>\"stdout\"}\n\n2020-02-29 00:06:26 +0000 [info]: #0 Timeout flush: kubernetes.var.log.containers.foo-57bd446b67-5htz9_api_service-8724f4d1297417ec0572007a48dbc381dbdfbf38e634ff63c460a600b2ec2bd2.log:default\n", "stream"=>"stdout"}
If you use k8s, you can try configuration for k8s in README.md. Please search k8s in README.md and check the sample.
k8s
Problem
Fluentd not sendings logs to es or in s3 successfully. I got a similar issue to https://github.com/fluent-plugins-nursery/fluent-plugin-concat/issues/37 ...
My configuration
Warning message