Can not start after restart k0scontroler, bufio.Scanner: token too long

wenerme commented 1 year ago

Before creating an issue, make sure you've checked the following:

[X] You are running the latest released version of k0s
[X] Make sure you've searched for existing issues, both open and closed
[X] Make sure you've searched for PRs too, a fix might've been merged already
[X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 5.15.87-0-lts #1-Alpine SMP Fri, 13 Jan 2023 09:15:31 +0000 x86_64 GNU/Linux
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.17.1
PRETTY_NAME="Alpine Linux v3.17"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

Version

v1.25.5+k0s.0

Sysinfo

`k0s sysinfo`

```text Machine ID: "X" (from machine) (pass) Total memory: 188.6 GiB (pass) Disk space available for /var/lib/k0s: 36.4 TiB (pass) Operating system: Linux (pass) Linux kernel release: 5.15.87-0-lts (pass) Max. file descriptors per process: current: 4096 / max: 4096 (warning: < 65536) Executable in path: modprobe: /sbin/modprobe (pass) /proc file system: mounted (0x9fa0) (pass) Control Groups: version 1 (pass) cgroup controller "cpu": available (pass) cgroup controller "cpuacct": available (pass) cgroup controller "cpuset": available (pass) cgroup controller "memory": available (pass) cgroup controller "devices": available (pass) cgroup controller "freezer": available (pass) cgroup controller "pids": available (pass) cgroup controller "hugetlb": available (pass) cgroup controller "blkio": available (pass) CONFIG_CGROUPS: Control Group support: no kernel config found (warning) CONFIG_NAMESPACES: Namespaces support: no kernel config found (warning) CONFIG_NET: Networking support: no kernel config found (warning) CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: no kernel config found (warning) CONFIG_PROC_FS: /proc file system support: no kernel config found (warning) ```

What happened?

After restart k0scontroller, the node can not becoma Ready, just keep restarting

time="2023-02-01 11:59:57" level=error msg="Error while reading from Writer: bufio.Scanner: token too long" component=kubelet
time="2023-02-01 11:59:57" level=warning msg="signal: broken pipe" component=kubelet
time="2023-02-01 11:59:57" level=info msg="respawning in 5s" component=kubelet

Have to reboot os to make k0scontroller start

sudo grep 'bufio.Scanner: token too long' /var/log/k0s.log | wc -l
4369

I have this problem since 1.23, now upgrade to 1.25, tested every version, have the same problem

Steps to reproduce

service k0scontroller restart

Expected behavior

No response

Actual behavior

No response

Screenshots and logs

No response

Additional context

using zfs
/var/lib/k0s is mounted zfs

findmnt /var/lib/k0s
TARGET       SOURCE       FSTYPE OPTIONS
/var/lib/k0s main/var/k0s zfs    rw,noatime,xattr,noacl

/etc/k0s/containerd.toml

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "zfs"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://fogjl973.mirror.aliyuncs.com", "https://8x40wsit.mirror.aliyuncs.com", "https://docker.mirrors.ustc.edu.cn", "https://registry-1.docker.io"]

/etc/k0s/k0s.yaml

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: X
spec:
  api:
    address: 192.168.66.110
    k0sApiPort: 9443
    port: 6443
    sans:
    - 192.168.66.110
    - 172.17.0.1
    - 127.0.0.1
    tunneledNetworkingMode: false
  storage:
    etcd:
      peerAddress: 192.168.66.110
    type: etcd
  controllerManager:
    extraArgs:
      cluster-name: X
  telemetry:
    enabled: false

I have a cron to start k0scontroller every month to make sure the cert rotation, do I have to do this for k0s too ?

/etc/periodic/monthly/kube-restart

#!/bin/sh -
sleep ${RANDOM:0:2}m

rc-service k3s restart --ifstarted
rc-service k0scontroller restart --ifstarted

BTW, k3s restart without problem.

kke commented 1 year ago

This may actually be an issue in kubelet's logging.

The string Error while reading from Writer comes from https://github.com/sirupsen/logrus/blob/master/writer.go#L63 :

        entry.Errorf("Error while reading from Writer: %s", err)

It uses the default bufio.Scanner MaxTokenSize which is 64kb. Apparently, kubelet is trying to log something that doesn't have a linefeed in a >64kb chunk.

Maybe it could be possible to work around this by modifying kubelet's logging config to some other format or to be less verbose through kubelet-extraargs?

wenerme commented 1 year ago

What I can do ? There are too many flags https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

k0sproject / k0s