fluentd failed to process log after Kubernetes upgrade from 1.19.x to 1.20.x

[hetian-github]

Our log system for business is EFK(elasticsearch+fluentd+kibana). The fluentd daemonset is running on AKS(Microsoft Azure Kubernetes service). I haven't done any change to the fluentd configuration. And after I upgrade the kubernetes from 1.19.x to 1.20.x, the fluentd has not sent log to Elasticsearch anymore. From the logs of fluentd I can see it did grab logs which are located in directory /var/log/containers on kubernetes node. But then it did not send the log data to Elasticsearch. And I can see that the buffer directory is empty. When it worked well, there were buffer files written to the buffer directory. Is there some change of Kubernetes 1.20.x that affect fluentd running?

[BenHorneIL]

What is the image of fluentd you are running and what do you have set for the conf?

we had to update

image: fluent/fluentd-kubernetes-daemonset:v1.4-debian-elasticsearch-1

to

image: fluent/fluentd-kubernetes-daemonset:v1.12-debian-elasticsearch6-1

As well as update the kubernetes.conf source:

format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days