kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.54k stars 1.57k forks source link

Increase the log rotation values for CI #3772

Closed aojea closed 2 weeks ago

aojea commented 3 weeks ago

The default values for log rotation on the kubelet

containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5),

are not enough to keep all the logs of a job

https://gcsweb.k8s.io/gcs/kubernetes-ci-logs/pr-logs/pull/128474/pull-kubernetes-e2e-kind/1852600889194319872/artifacts/kind-control-plane/pods/kube-system_kube-apiserver-kind-control-plane_88820ae9448fa5cfa46a5444a7bf898e/kube-apiserver/

This makes hard to debug or troubleshoot CI problems, specially rare flakes that happen very infrequently, since it is impossible to understand what happened without logs.

We need to use some values that allow to store all the logs that happen during the entire job duration

I think that increasing the values to:

container-log-max-files: "10"
container-log-max-size: "5Mi"

The place to change them is in https://github.com/kubernetes-sigs/kind/blob/4f444f8afa344eb2de27e68a2a24b823301fc8d9/hack/ci/e2e-k8s.sh#L101-L124

https://knowledge.broadcom.com/external/article/327480/customizing-the-kubelet-parameters-for-l.html

aojea commented 3 weeks ago

/help /priority important-soon

k8s-ci-robot commented 3 weeks ago

@aojea: This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes-sigs/kind/issues/3772): >/help >/priority important-soon Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
elieser1101 commented 3 weeks ago

/assign