search

kubernetes / kubelet

kubelet component configs
Apache License 2.0
317 stars 143 forks source link

Clarity required around Logging Architecture of Kubelet. #51

Open javadevmtl opened 3 months ago

javadevmtl commented 3 months ago

Running Kubernetes 1.27.6

I have some performance issues regarding logrotation where I miss logs, I'm not here blaming Kubernetes, but there needs some clarification around the Log Architecture documents here Loggin Architecture or more details.

We have set containerLogMaxSize and containerLogMaxFiles which works fine and is rotating logs as per the settings. We are trying to debug an issue where we miss logs when there is large volume.

Now here is the disparity.

  1. Based on the linked document above and the 2 settings we are lead to believe that Kubelet is doing the rotation, while many online resources, forums, seem to be saying it's logrotate.
  2. Looking at the /var/log/pods/ folder indeed the logs are being rotate as per the 2 settings, but they are also gzipped. Nowhere does the document explain this and there's no settings indicated anywhere for compressing the logs. Which leads us to belive that the file is being zipped before the file agent can finish reading all the contents of the file.
  3. If logrotate is truly used does Kubelet pass these 2 settings containerLogMaxSize and containerLogMaxFiles to logrotate somehow and if so how (We looked in /etc/logrotate.d/ and /etc/logrotate.conf and found no indication of a Kubenrnetes config pointing to /var/log/pods/)?
k8s-triage-robot commented 3 weeks ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale