Closed veenabenoy closed 4 years ago
PR would be welcome ;-)
I followed another approach.
configMaps:
useDefaults:
systemConf: true
containersInputConf: false
systemInputConf: true
forwardInputConf: true
monitoringConf: true
outputConf: true
And I verified refresh interval is updated in fluentd configmaps after deploying the helm chart
(using kubectl describe configmaps <configmap-name>
The draw back is just for updating one flag 'refresh_interval" I had to define entire section of containers.input.conf inorder to pass it as extra config parameter
extraConfigMaps:
containers.input.conf: |-
<source>
@id fluentd-containers.log
@type tail
path /var/log/containers/*.log
pos_file /var/log/containers.log.pos
tag raw.kubernetes.*
refresh_interval 3s
read_from_head true
<parse>
@type multi_format <pattern>
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%NZ
</pattern>
<pattern>
format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
time_format %Y-%m-%dT%H:%M:%S.%N%:z
</pattern>
</parse>
</source>
# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
@id raw.kubernetes
@type detect_exceptions
remove_tag_prefix raw
message log
stream stream
multiline_flush_interval 5
max_bytes 500000
max_lines 1000
</match>
# Concatenate multi-line logs
<filter **>
@id filter_concat
@type concat
key message
multiline_end_regexp /\n$/
separator ""
timeout_label @NORMAL
flush_interval 5
</filter>
# Enriches records with Kubernetes metadata
<filter kubernetes.**>
@id filter_kubernetes_metadata
@type kubernetes_metadata
</filter>
# Fixes json fields in Elasticsearch
<filter kubernetes.**>
@id filter_parser
@type parser
key_name log
reserve_time true
reserve_data true
remove_key_name_field true
<parse>
@type multi_format
<pattern>
format json
</pattern>
<pattern>
format none
</pattern>
</parse>
</filter>
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
We are using fluentd, installed in the cluster using helm chart kiwigrid/fluentd-elasticsearch --version 6.1.1 we set elasticsearch.host as part of helm install command. So all other fluent id configuration take default values.
We have some pods in the system which will execute in less than 60sec. We observed that Fluentd is not capturing the logs of such short-life pods.We assume this is happening because fluentd by default is configured(refresh_interval) to update the list of log files it monitors only once a minute. Due to the very short run time of the pods, they are already deleted when fluentd checks next time.
We tried manually adding this value in a configmap on a running system and it solved the problem (restart the fluentd pods after adding config map). Unfortunately, this variable is not exposed in the kiwigrid helm chart. So creating a request to add it part of configurable parameteres of Fluentd elasticsearch chart
helm_binary_version: 2.16.1 Kubernetes - v1.15.3