search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.86k stars 2.33k forks source link

Repeated "failed to fetch hugetlb info" error from runc on Raspberry Pi #4580

Closed alexellis closed 2 years ago

alexellis commented 2 years ago

I'm seeing lots of errors with K3s and Raspberry Pi Bullseye. K3s does start and kubectl does respond.

Environmental Info: K3s Version: 

pi@raspberrypi:~ $ k3s -v
k3s version v1.21.6+k3s1 (df033fa2)
go version go1.16.8

Node(s) CPU architecture, OS, and Version: 

pi@raspberrypi:~ $ sudo kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6+k3s1", GitCommit:"df033fa248bc2c9f636e4c0ff2b782cb8edbbf10", GitTreeState:"clean", BuildDate:"2021-11-04T00:25:17Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6+k3s1", GitCommit:"df033fa248bc2c9f636e4c0ff2b782cb8edbbf10", GitTreeState:"clean", BuildDate:"2021-11-04T00:25:17Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/arm"}
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.63-v7l+ #1459 SMP Wed Oct 6 16:41:57 BST 2021 armv7l GNU/Linux
pi@raspberrypi:~ $ cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@raspberrypi:~ $ 

pi@raspberrypi:~ $ cat /boot/cmdline.txt 
console=serial0,115200 console=tty1 root=PARTUUID=ed53b1f6-02 rootfstype=ext4 fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Cluster Configuration:

Describe the bug:

Can the errors be resolved?

Steps To Reproduce:

Expected behavior:

No errors

Actual behavior:

See below

Additional context / logs:

pi@raspberrypi:~ $ sudo journalctl -u k3s -f
-- Journal begins at Sat 2021-10-30 12:29:41 BST. --
Nov 25 14:40:09 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:40:09 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:40:09 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:09 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:09 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:09 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:09 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:09 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:09 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:09 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:11 raspberrypi k3s[662]: E1125 14:40:11.825989     662 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d\": RecentStats: unable to find data in memory cache], [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache], [\"/kubepods\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/4724f2bebd6143ce5e5bb3d77495e807b700337cd5714e3136d328f6d501db45\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/081892bdffaf1e5ef168991de0c0d79aa3d104d9663ce136d8b2d692ff9312c6\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/3c5a5d1c5d81fd0224de5d272b13f8dbc4c1b951fc5e5e03032bf2ccb5069b18\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/e66b33af61ae89800a904f8202a797fd3b2816f4285749718dc428cfa41544e0\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a\": RecentStats: unable to find data in memory cache]"
Nov 25 14:40:13 raspberrypi k3s[662]: E1125 14:40:13.002665     662 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/k3s.service\": failed to get container info for \"/system.slice/k3s.service\": partial failures: [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache]" containerName="/system.slice/k3s.service"
Nov 25 14:40:13 raspberrypi k3s[662]: E1125 14:40:13.004853     662 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubepods\": failed to get container info for \"/kubepods\": partial failures: [\"/kubepods\": RecentStats: unable to find data in memory cache]" containerName="/kubepods"
Nov 25 14:40:13 raspberrypi k3s[662]: E1125 14:40:13.004978     662 helpers.go:673] "Eviction manager: failed to construct signal" err="system container \"pods\" not found in metrics" signal=allocatableMemory.available
Nov 25 14:40:14 raspberrypi k3s[662]: W1125 14:40:14.124483     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:14 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:14 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:14 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:14 raspberrypi k3s[662]: W1125 14:40:14.919929     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:14 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:14 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:14 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:14 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:14 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:15 raspberrypi k3s[662]: W1125 14:40:15.602128     662 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Nov 25 14:40:15 raspberrypi k3s[662]: W1125 14:40:15.985805     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:15 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:15 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:15 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:15 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:15 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:15 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:15 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:15 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:15 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:15 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:15 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:15 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:15 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:15 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:15 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:15 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:15 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:23 raspberrypi k3s[662]: E1125 14:40:23.084443     662 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/3c5a5d1c5d81fd0224de5d272b13f8dbc4c1b951fc5e5e03032bf2ccb5069b18\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/081892bdffaf1e5ef168991de0c0d79aa3d104d9663ce136d8b2d692ff9312c6\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e\": RecentStats: unable to find data in memory cache], [\"/kubepods\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/e66b33af61ae89800a904f8202a797fd3b2816f4285749718dc428cfa41544e0\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/4724f2bebd6143ce5e5bb3d77495e807b700337cd5714e3136d328f6d501db45\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort\": RecentStats: unable to find data in memory cache], [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43\": RecentStats: unable to find data in memory cache]"
Nov 25 14:40:23 raspberrypi k3s[662]: W1125 14:40:23.511919     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:23 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:23 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:23 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:23 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:23 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:23 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:23 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:23 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:23 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:23 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:23 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:23 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:23 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:23 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:23 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:23 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:23 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:24 raspberrypi k3s[662]: E1125 14:40:24.267370     662 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/k3s.service\": failed to get container info for \"/system.slice/k3s.service\": partial failures: [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache]" containerName="/system.slice/k3s.service"
Nov 25 14:40:24 raspberrypi k3s[662]: E1125 14:40:24.269398     662 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubepods\": failed to get container info for \"/kubepods\": partial failures: [\"/kubepods\": RecentStats: unable to find data in memory cache]" containerName="/kubepods"
Nov 25 14:40:24 raspberrypi k3s[662]: E1125 14:40:24.269554     662 helpers.go:673] "Eviction manager: failed to construct signal" err="system container \"pods\" not found in metrics" signal=allocatableMemory.available
Nov 25 14:40:29 raspberrypi k3s[662]: W1125 14:40:29.642138     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:29 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:29 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:29 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:29 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:29 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:29 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:29 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:29 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:29 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:29 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:29 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:29 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:29 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:29 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:29 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:29 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:29 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:30 raspberrypi k3s[662]: W1125 14:40:30.911050     662 container.go:586] Failed to update stats for container "/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:30 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:30 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:30 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:30 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:30 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:30 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:30 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:30 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:30 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:30 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:30 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:30 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:30 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:30 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:30 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:30 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:30 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:33 raspberrypi k3s[662]: W1125 14:40:33.747738     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:33 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:33 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:33 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:33 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:33 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:33 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:33 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:33 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:33 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:33 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:33 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:33 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:33 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:33 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:33 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:33 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:33 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:34 raspberrypi k3s[662]: E1125 14:40:34.300204     662 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/4724f2bebd6143ce5e5bb3d77495e807b700337cd5714e3136d328f6d501db45\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9\": RecentStats: unable to find data in memory cache], [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/081892bdffaf1e5ef168991de0c0d79aa3d104d9663ce136d8b2d692ff9312c6\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/e66b33af61ae89800a904f8202a797fd3b2816f4285749718dc428cfa41544e0\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/3c5a5d1c5d81fd0224de5d272b13f8dbc4c1b951fc5e5e03032bf2ccb5069b18\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d\": RecentStats: unable to find data in memory cache], [\"/kubepods\": RecentStats: unable to find data in memory cache]"
Nov 25 14:40:35 raspberrypi k3s[662]: W1125 14:40:35.140644     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:35 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:35 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:35 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:35 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:35 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:35 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:35 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:35 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:35 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:35 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:35 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:35 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:35 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:35 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:35 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:35 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:35 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:35 raspberrypi k3s[662]: E1125 14:40:35.781858     662 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/k3s.service\": failed to get container info for \"/system.slice/k3s.service\": partial failures: [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache]" containerName="/system.slice/k3s.service"
Nov 25 14:40:35 raspberrypi k3s[662]: E1125 14:40:35.783127     662 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubepods\": failed to get container info for \"/kubepods\": partial failures: [\"/kubepods\": RecentStats: unable to find data in memory cache]" containerName="/kubepods"
Nov 25 14:40:35 raspberrypi k3s[662]: E1125 14:40:35.783281     662 helpers.go:673] "Eviction manager: failed to construct signal" err="system container \"pods\" not found in metrics" signal=allocatableMemory.available
Nov 25 14:40:39 raspberrypi k3s[662]: W1125 14:40:39.357058     662 container.go:586] Failed to update stats for container "/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/081892bdffaf1e5ef168991de0c0d79aa3d104d9663ce136d8b2d692ff9312c6": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:39 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:39 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:39 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:39 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:39 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:39 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:39 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:39 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:40:39 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:40:39 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:39 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:39 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:39 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:39 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:39 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:39 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:39 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:40:40 raspberrypi k3s[662]: E1125 14:40:40.744228     662 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/3c5a5d1c5d81fd0224de5d272b13f8dbc4c1b951fc5e5e03032bf2ccb5069b18\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/e66b33af61ae89800a904f8202a797fd3b2816f4285749718dc428cfa41544e0\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee\": RecentStats: unable to find data in memory cache], [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable\": RecentStats: unable to find data in memory cache], [\"/kubepods\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/4724f2bebd6143ce5e5bb3d77495e807b700337cd5714e3136d328f6d501db45\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/081892bdffaf1e5ef168991de0c0d79aa3d104d9663ce136d8b2d692ff9312c6\": RecentStats: unable to find data in memory cache]"
Nov 25 14:40:40 raspberrypi k3s[662]: E1125 14:40:40.744612     662 summary_sys_containers.go:82] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubepods\": failed to get container info for \"/kubepods\": partial failures: [\"/kubepods\": RecentStats: unable to find data in memory cache]" containerName="/kubepods"
Nov 25 14:40:40 raspberrypi k3s[662]: E1125 14:40:40.744709     662 summary_sys_containers.go:82] "Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/k3s.service\": failed to get container info for \"/system.slice/k3s.service\": partial failures: [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache]" containerName="/system.slice/k3s.service"
Nov 25 14:40:42 raspberrypi k3s[662]: W1125 14:40:42.528916     662 container.go:586] Failed to update stats for container "/system.slice/k3s.service": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:40:42 raspberrypi k3s[662]: failed to fetch hugetlb info
Nov 25 14:40:42 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:40:42 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:40:42 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:40:42 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:40:42 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:40:42 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:40:42 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:40:42 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:40:42 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:40:42 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:40:42 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:40:42 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:40:42 raspberrypi k3s[662]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:40:42 raspberrypi k3s[662]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:40:42 raspberrypi k3s[662]: runtime.goexit
Nov 25 14:40:42 raspberrypi k3s[662]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
^C
alexellis commented 2 years ago

Start-up logs. Sorry it's a lot of output

Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.012079396Z" level=info msg="Starting k3s v1.21.6+k3s1 (df033fa2)"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.012862312Z" level=info msg="Cluster bootstrap already complete"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.068206266Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.068862702Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.069789117Z" level=info msg="Database tables and indexes are up to date"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.087638208Z" level=info msg="Kine listening on unix://kine.sock"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.089034524Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Nov 25 14:42:07 raspberrypi k3s[8346]: Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.096656    8346 server.go:656] external host was not specified, using 192.168.0.24
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.098347    8346 server.go:195] Version: v1.21.6+k3s1
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.124460542Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.128507    8346 shared_informer.go:240] Waiting for caches to sync for node_authorizer
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.129695940Z" level=info msg="Waiting for API server to become available"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.131187995Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.134238808Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.140299751Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.141194721Z" level=info msg="To join node to cluster: k3s agent -s https://192.168.0.24:6443 -t ${NODE_TOKEN}"
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.140766    8346 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.142974    8346 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.148025673Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.148870625Z" level=info msg="Run: k3s kubectl"
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.150420    8346 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.153348    8346 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.280072    8346 instance.go:283] Using reconciler: lease
Nov 25 14:42:07 raspberrypi k3s[8346]: I1125 14:42:07.455664    8346 rest.go:130] the default service ipfamily for this cluster is: IPv4
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.557129579Z" level=info msg="certificate CN=raspberrypi signed by CN=k3s-server-ca@1637849021: notBefore=2021-11-25 14:03:41 +0000 UTC notAfter=2022-11-25 14:42:07 +0000 UTC"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.582595834Z" level=info msg="certificate CN=system:node:raspberrypi,O=system:nodes signed by CN=k3s-client-ca@1637849021: notBefore=2021-11-25 14:03:41 +0000 UTC notAfter=2022-11-25 14:42:07 +0000 UTC"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.824302895Z" level=info msg="Module overlay was already loaded"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.824392783Z" level=info msg="Module nf_conntrack was already loaded"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.824441319Z" level=info msg="Module br_netfilter was already loaded"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.824489837Z" level=info msg="Module iptable_nat was already loaded"
Nov 25 14:42:07 raspberrypi k3s[8346]: W1125 14:42:07.824619    8346 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.848426389Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Nov 25 14:42:07 raspberrypi k3s[8346]: time="2021-11-25T14:42:07.849986999Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Nov 25 14:42:08 raspberrypi k3s[8346]: W1125 14:42:08.729997    8346 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Nov 25 14:42:08 raspberrypi k3s[8346]: W1125 14:42:08.778287    8346 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Nov 25 14:42:08 raspberrypi k3s[8346]: W1125 14:42:08.811324    8346 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Nov 25 14:42:08 raspberrypi k3s[8346]: W1125 14:42:08.840647    8346 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Nov 25 14:42:08 raspberrypi k3s[8346]: W1125 14:42:08.851837    8346 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
Nov 25 14:42:08 raspberrypi k3s[8346]: time="2021-11-25T14:42:08.856906156Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused\""
Nov 25 14:42:08 raspberrypi k3s[8346]: W1125 14:42:08.876617    8346 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
Nov 25 14:42:08 raspberrypi k3s[8346]: W1125 14:42:08.876674    8346 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
Nov 25 14:42:08 raspberrypi k3s[8346]: I1125 14:42:08.917764    8346 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Nov 25 14:42:08 raspberrypi k3s[8346]: I1125 14:42:08.917821    8346 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Nov 25 14:42:09 raspberrypi k3s[8346]: time="2021-11-25T14:42:09.860681233Z" level=info msg="Containerd is now running"
Nov 25 14:42:09 raspberrypi k3s[8346]: time="2021-11-25T14:42:09.932447797Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Nov 25 14:42:09 raspberrypi k3s[8346]: time="2021-11-25T14:42:09.960515835Z" level=info msg="Handling backend connection request [raspberrypi]"
Nov 25 14:42:09 raspberrypi k3s[8346]: time="2021-11-25T14:42:09.963090673Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/1e69816ee256b4bf0342f5188d55d937b9b7ada03be18500c0dd943b9ac0fff4/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=raspberrypi --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Nov 25 14:42:09 raspberrypi k3s[8346]: time="2021-11-25T14:42:09.964903280Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=raspberrypi --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Nov 25 14:42:09 raspberrypi k3s[8346]: W1125 14:42:09.965791    8346 server.go:224] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Nov 25 14:42:09 raspberrypi k3s[8346]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Nov 25 14:42:09 raspberrypi k3s[8346]: Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Nov 25 14:42:09 raspberrypi k3s[8346]: Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Nov 25 14:42:09 raspberrypi k3s[8346]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Nov 25 14:42:10 raspberrypi k3s[8346]: I1125 14:42:10.139025    8346 server.go:436] "Kubelet version" kubeletVersion="v1.21.6+k3s1"
Nov 25 14:42:10 raspberrypi k3s[8346]: E1125 14:42:10.337862    8346 node.go:161] Failed to retrieve node info: nodes "raspberrypi" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Nov 25 14:42:10 raspberrypi k3s[8346]: I1125 14:42:10.379916    8346 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
Nov 25 14:42:10 raspberrypi k3s[8346]: W1125 14:42:10.380754    8346 manager.go:159] Cannot detect current cgroup on cgroup v2
Nov 25 14:42:11 raspberrypi k3s[8346]: E1125 14:42:11.421707    8346 node.go:161] Failed to retrieve node info: nodes "raspberrypi" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Nov 25 14:42:13 raspberrypi k3s[8346]: E1125 14:42:13.831723    8346 node.go:161] Failed to retrieve node info: nodes "raspberrypi" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.390845    8346 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.398615    8346 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.399487    8346 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.399721    8346 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.399793    8346 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.399833    8346 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.399861    8346 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.404039    8346 kubelet.go:404] "Attempting to sync node with API server"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.404118    8346 kubelet.go:272] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.404215    8346 kubelet.go:283] "Adding apiserver pod source"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.404285    8346 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.410492    8346 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.11-k3s1" apiVersion="v1alpha2"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.411504    8346 server.go:1191] "Started kubelet"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.432876    8346 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.433735    8346 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
Nov 25 14:42:15 raspberrypi k3s[8346]: E1125 14:42:15.455008    8346 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
Nov 25 14:42:15 raspberrypi k3s[8346]: E1125 14:42:15.456075    8346 kubelet.go:1306] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.468567    8346 volume_manager.go:271] "Starting Kubelet Volume Manager"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.470349    8346 server.go:409] "Adding debug handlers to kubelet server"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.477659    8346 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.572019    8346 kuberuntime_manager.go:1044] "Updating runtime config through cri with podcidr" CIDR="10.42.0.0/24"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.609323    8346 kubelet_network.go:76] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.42.0.0/24"
Nov 25 14:42:15 raspberrypi k3s[8346]: I1125 14:42:15.610209    8346 kubelet_node_status.go:71] "Attempting to register node" node="raspberrypi"
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.753425    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.772310    8346 container.go:586] Failed to update stats for container "/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/4724f2bebd6143ce5e5bb3d77495e807b700337cd5714e3136d328f6d501db45": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.778974    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/e66b33af61ae89800a904f8202a797fd3b2816f4285749718dc428cfa41544e0": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.780779    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.782167    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.809373    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.811286    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.812984    8346 container.go:586] Failed to update stats for container "/kubepods": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.821856    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.826109    8346 container.go:586] Failed to update stats for container "/kubepods/burstable": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.837248    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.838512    8346 container.go:586] Failed to update stats for container "/system.slice/k3s.service": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.862562    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.867326    8346 container.go:586] Failed to update stats for container "/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.878493    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.890179    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/3c5a5d1c5d81fd0224de5d272b13f8dbc4c1b951fc5e5e03032bf2ccb5069b18": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.897602    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.905662    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:15 raspberrypi k3s[8346]: W1125 14:42:15.912071    8346 container.go:586] Failed to update stats for container "/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:15 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:15 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:15 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:15 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:15 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.304392    8346 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.425035    8346 apiserver.go:52] "Watching apiserver"
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.510761    8346 cpu_manager.go:199] "Starting CPU manager" policy="none"
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.510858    8346 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.510927    8346 state_mem.go:36] "Initialized new in-memory state store"
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.512115    8346 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.512197    8346 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.512232    8346 policy_none.go:44] "None policy: Start"
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.516010    8346 manager.go:600] "Failed to retrieve checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Nov 25 14:42:16 raspberrypi k3s[8346]: I1125 14:42:16.520454    8346 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
Nov 25 14:42:16 raspberrypi k3s[8346]: E1125 14:42:16.620126    8346 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/e66b33af61ae89800a904f8202a797fd3b2816f4285749718dc428cfa41544e0\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5\": RecentStats: unable to find data in memory cache], [\"/kubepods\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/4724f2bebd6143ce5e5bb3d77495e807b700337cd5714e3136d328f6d501db45\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort\": RecentStats: unable to find data in memory cache], [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/3c5a5d1c5d81fd0224de5d272b13f8dbc4c1b951fc5e5e03032bf2ccb5069b18\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable\": RecentStats: unable to find data in memory cache]"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.125019    8346 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.125653    8346 status_manager.go:157] "Starting to sync pod status with apiserver"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.142047    8346 kubelet.go:1846] "Starting kubelet main sync loop"
Nov 25 14:42:17 raspberrypi k3s[8346]: E1125 14:42:17.163994    8346 kubelet.go:1870] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 25 14:42:17 raspberrypi k3s[8346]: E1125 14:42:17.204992    8346 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/k3s.service\": failed to get container info for \"/system.slice/k3s.service\": partial failures: [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache]" containerName="/system.slice/k3s.service"
Nov 25 14:42:17 raspberrypi k3s[8346]: E1125 14:42:17.208820    8346 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubepods\": failed to get container info for \"/kubepods\": partial failures: [\"/kubepods\": RecentStats: unable to find data in memory cache]" containerName="/kubepods"
Nov 25 14:42:17 raspberrypi k3s[8346]: E1125 14:42:17.209920    8346 helpers.go:673] "Eviction manager: failed to construct signal" err="system container \"pods\" not found in metrics" signal=allocatableMemory.available
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.264964    8346 topology_manager.go:187] "Topology Admit Handler"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.265525    8346 topology_manager.go:187] "Topology Admit Handler"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.265862    8346 topology_manager.go:187] "Topology Admit Handler"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.266262    8346 topology_manager.go:187] "Topology Admit Handler"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.266615    8346 topology_manager.go:187] "Topology Admit Handler"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.267160    8346 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="c5f8d741b7da1860d32f079991fae048dcd413fa36613839f682aa8f57036166"
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.267351    8346 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="dc3660c471f0d6d749fc2f5db0df565ae3da008305be69e068143eeadf78c0b8"
Nov 25 14:42:17 raspberrypi k3s[8346]: E1125 14:42:17.302714    8346 reflector.go:138] object-"kube-system"/"local-path-config": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "local-path-config" is forbidden: User "system:node:raspberrypi" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object
Nov 25 14:42:17 raspberrypi k3s[8346]: E1125 14:42:17.304640    8346 reflector.go:138] object-"kube-system"/"coredns": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:raspberrypi" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.304937    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5fvl\" (UniqueName: \"kubernetes.io/projected/49ff49df-45fe-41f5-ad0e-3f1220f65d37-kube-api-access-s5fvl\") pod \"local-path-provisioner-5ff76fc89d-z8gh9\" (UID: \"49ff49df-45fe-41f5-ad0e-3f1220f65d37\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305062    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb4ae87c-ea92-4bdf-92d7-c05298ddb138-tmp-dir\") pod \"metrics-server-86cbb8457f-jllbk\" (UID: \"eb4ae87c-ea92-4bdf-92d7-c05298ddb138\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305174    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwk9t\" (UniqueName: \"kubernetes.io/projected/eb4ae87c-ea92-4bdf-92d7-c05298ddb138-kube-api-access-gwk9t\") pod \"metrics-server-86cbb8457f-jllbk\" (UID: \"eb4ae87c-ea92-4bdf-92d7-c05298ddb138\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305253    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfncz\" (UniqueName: \"kubernetes.io/projected/57269015-f305-405d-afa5-6ba6c8f645b5-kube-api-access-vfncz\") pod \"coredns-7448499f4d-msgmv\" (UID: \"57269015-f305-405d-afa5-6ba6c8f645b5\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305357    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e34c2e70-3667-4097-b5b0-052ffdac65d9-data\") pod \"traefik-97b44b794-rdhzh\" (UID: \"e34c2e70-3667-4097-b5b0-052ffdac65d9\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305465    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzzz\" (UniqueName: \"kubernetes.io/projected/fa84d7db-7dd6-4247-9c0f-0ddba859b8d4-kube-api-access-fwzzz\") pod \"svclb-traefik-lcklq\" (UID: \"fa84d7db-7dd6-4247-9c0f-0ddba859b8d4\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305569    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff49df-45fe-41f5-ad0e-3f1220f65d37-config-volume\") pod \"local-path-provisioner-5ff76fc89d-z8gh9\" (UID: \"49ff49df-45fe-41f5-ad0e-3f1220f65d37\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305654    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57269015-f305-405d-afa5-6ba6c8f645b5-config-volume\") pod \"coredns-7448499f4d-msgmv\" (UID: \"57269015-f305-405d-afa5-6ba6c8f645b5\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305759    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34c2e70-3667-4097-b5b0-052ffdac65d9-tmp\") pod \"traefik-97b44b794-rdhzh\" (UID: \"e34c2e70-3667-4097-b5b0-052ffdac65d9\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305854    8346 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvmpk\" (UniqueName: \"kubernetes.io/projected/e34c2e70-3667-4097-b5b0-052ffdac65d9-kube-api-access-pvmpk\") pod \"traefik-97b44b794-rdhzh\" (UID: \"e34c2e70-3667-4097-b5b0-052ffdac65d9\") "
Nov 25 14:42:17 raspberrypi k3s[8346]: I1125 14:42:17.305887    8346 reconciler.go:157] "Reconciler: start to sync state"
Nov 25 14:42:17 raspberrypi k3s[8346]: E1125 14:42:17.317317    8346 reflector.go:138] object-"kube-system"/"kube-root-ca.crt": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kube-root-ca.crt" is forbidden: User "system:node:raspberrypi" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.303309    8346 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.303353    8346 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.304824    8346 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.307874    8346 secure_serving.go:202] Serving securely on 127.0.0.1:6444
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.307937    8346 tlsconfig.go:240] Starting DynamicServingCertificateController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.308016    8346 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.308300    8346 autoregister_controller.go:141] Starting autoregister controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.308444    8346 cache.go:32] Waiting for caches to sync for autoregister controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.308719    8346 customresource_discovery_controller.go:209] Starting DiscoveryController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.309436    8346 crdregistration_controller.go:111] Starting crd-autoregister controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.309826    8346 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.310241    8346 controller.go:86] Starting OpenAPI controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.310308    8346 naming_controller.go:291] Starting NamingConditionController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.310378    8346 establishing_controller.go:76] Starting EstablishingController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.310585    8346 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.311191    8346 node.go:161] Failed to retrieve node info: nodes "raspberrypi" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.311671    8346 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.311760    8346 crd_finalizer.go:266] Starting CRDFinalizer
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.312880    8346 available_controller.go:475] Starting AvailableConditionController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.312927    8346 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.312983    8346 controller.go:83] Starting OpenAPI AggregationController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.310000    8346 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.315736    8346 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.327596    8346 apiservice_controller.go:97] Starting APIServiceRegistrationController
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.327640    8346 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.328423    8346 apf_controller.go:307] Starting API Priority and Fairness config controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.328636    8346 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.328701    8346 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.338021    8346 reflector.go:138] object-"kube-system"/"local-path-config": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "local-path-config" is forbidden: User "system:node:raspberrypi" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.409924    8346 configmap.go:200] Couldn't get configMap kube-system/coredns: failed to sync configmap cache: timed out waiting for the condition
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.410188    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/57269015-f305-405d-afa5-6ba6c8f645b5-config-volume podName:57269015-f305-405d-afa5-6ba6c8f645b5 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:18.91010854 +0000 GMT m=+12.569008669 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57269015-f305-405d-afa5-6ba6c8f645b5-config-volume\") pod \"coredns-7448499f4d-msgmv\" (UID: \"57269015-f305-405d-afa5-6ba6c8f645b5\") : failed to sync configmap cache: timed out waiting for the condition"
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.422624    8346 projected.go:293] Couldn't get configMap kube-system/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.422750    8346 projected.go:199] Error preparing data for projected volume kube-api-access-pvmpk for pod kube-system/traefik-97b44b794-rdhzh: [failed to fetch token: serviceaccounts "traefik" is forbidden: User "system:node:raspberrypi" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object, failed to sync configmap cache: timed out waiting for the condition]
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.423080    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/projected/e34c2e70-3667-4097-b5b0-052ffdac65d9-kube-api-access-pvmpk podName:e34c2e70-3667-4097-b5b0-052ffdac65d9 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:18.922963751 +0000 GMT m=+12.581863916 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"kube-api-access-pvmpk\" (UniqueName: \"kubernetes.io/projected/e34c2e70-3667-4097-b5b0-052ffdac65d9-kube-api-access-pvmpk\") pod \"traefik-97b44b794-rdhzh\" (UID: \"e34c2e70-3667-4097-b5b0-052ffdac65d9\") : [failed to fetch token: serviceaccounts \"traefik\" is forbidden: User \"system:node:raspberrypi\" cannot create resource \"serviceaccounts/token\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'raspberrypi' and this object, failed to sync configmap cache: timed out waiting for the condition]"
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.445477    8346 projected.go:293] Couldn't get configMap kube-system/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.445574    8346 projected.go:199] Error preparing data for projected volume kube-api-access-s5fvl for pod kube-system/local-path-provisioner-5ff76fc89d-z8gh9: [failed to fetch token: serviceaccounts "local-path-provisioner-service-account" is forbidden: User "system:node:raspberrypi" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object, failed to sync configmap cache: timed out waiting for the condition]
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.445858    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/projected/49ff49df-45fe-41f5-ad0e-3f1220f65d37-kube-api-access-s5fvl podName:49ff49df-45fe-41f5-ad0e-3f1220f65d37 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:18.945758411 +0000 GMT m=+12.604658466 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"kube-api-access-s5fvl\" (UniqueName: \"kubernetes.io/projected/49ff49df-45fe-41f5-ad0e-3f1220f65d37-kube-api-access-s5fvl\") pod \"local-path-provisioner-5ff76fc89d-z8gh9\" (UID: \"49ff49df-45fe-41f5-ad0e-3f1220f65d37\") : [failed to fetch token: serviceaccounts \"local-path-provisioner-service-account\" is forbidden: User \"system:node:raspberrypi\" cannot create resource \"serviceaccounts/token\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'raspberrypi' and this object, failed to sync configmap cache: timed out waiting for the condition]"
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.542432    8346 shared_informer.go:247] Caches are synced for node_authorizer
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.549799    8346 configmap.go:200] Couldn't get configMap kube-system/local-path-config: failed to sync configmap cache: timed out waiting for the condition
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.553083    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/49ff49df-45fe-41f5-ad0e-3f1220f65d37-config-volume podName:49ff49df-45fe-41f5-ad0e-3f1220f65d37 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:19.052936134 +0000 GMT m=+12.711836373 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff49df-45fe-41f5-ad0e-3f1220f65d37-config-volume\") pod \"local-path-provisioner-5ff76fc89d-z8gh9\" (UID: \"49ff49df-45fe-41f5-ad0e-3f1220f65d37\") : failed to sync configmap cache: timed out waiting for the condition"
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.553676    8346 projected.go:293] Couldn't get configMap kube-system/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.554344    8346 projected.go:199] Error preparing data for projected volume kube-api-access-gwk9t for pod kube-system/metrics-server-86cbb8457f-jllbk: [failed to fetch token: serviceaccounts "metrics-server" is forbidden: User "system:node:raspberrypi" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object, failed to sync configmap cache: timed out waiting for the condition]
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.555026    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/projected/eb4ae87c-ea92-4bdf-92d7-c05298ddb138-kube-api-access-gwk9t podName:eb4ae87c-ea92-4bdf-92d7-c05298ddb138 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:19.054900832 +0000 GMT m=+12.713800923 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"kube-api-access-gwk9t\" (UniqueName: \"kubernetes.io/projected/eb4ae87c-ea92-4bdf-92d7-c05298ddb138-kube-api-access-gwk9t\") pod \"metrics-server-86cbb8457f-jllbk\" (UID: \"eb4ae87c-ea92-4bdf-92d7-c05298ddb138\") : [failed to fetch token: serviceaccounts \"metrics-server\" is forbidden: User \"system:node:raspberrypi\" cannot create resource \"serviceaccounts/token\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'raspberrypi' and this object, failed to sync configmap cache: timed out waiting for the condition]"
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.559611    8346 shared_informer.go:247] Caches are synced for crd-autoregister
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.589632    8346 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.613940    8346 cache.go:39] Caches are synced for AvailableConditionController controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.616370    8346 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.630507    8346 apf_controller.go:312] Running API Priority and Fairness config worker
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.650947    8346 cache.go:39] Caches are synced for autoregister controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.651149    8346 cache.go:39] Caches are synced for APIServiceRegistrationController controller
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.692503    8346 trace.go:205] Trace[799440604]: "GuaranteedUpdate etcd3" type:*core.Pod (25-Nov-2021 14:42:17.345) (total time: 1347ms):
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[799440604]: ---"Transaction prepared" 1213ms (14:42:00.559)
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[799440604]: ---"Transaction committed" 132ms (14:42:00.692)
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[799440604]: [1.347174848s] [1.347174848s] END
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.695012    8346 trace.go:205] Trace[1149092711]: "Patch" url:/api/v1/namespaces/kube-system/pods/local-path-provisioner-5ff76fc89d-z8gh9/status,user-agent:k3s/v1.21.6+k3s1 (linux/arm) kubernetes/df033fa,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (25-Nov-2021 14:42:17.344) (total time: 1350ms):
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1149092711]: ---"Object stored in database" 1323ms (14:42:00.694)
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1149092711]: [1.350071829s] [1.350071829s] END
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.724968    8346 trace.go:205] Trace[1793977998]: "GuaranteedUpdate etcd3" type:*coordination.Lease (25-Nov-2021 14:42:15.765) (total time: 2959ms):
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1793977998]: ---"Transaction prepared" 2824ms (14:42:00.589)
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1793977998]: ---"Transaction committed" 135ms (14:42:00.724)
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1793977998]: [2.959728501s] [2.959728501s] END
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.746880    8346 trace.go:205] Trace[17185194]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/raspberrypi,user-agent:k3s/v1.21.6+k3s1 (linux/arm) kubernetes/df033fa,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (25-Nov-2021 14:42:15.764) (total time: 2982ms):
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[17185194]: ---"Object stored in database" 2981ms (14:42:00.746)
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[17185194]: [2.982255257s] [2.982255257s] END
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.798491    8346 projected.go:199] Error preparing data for projected volume kube-api-access-fwzzz for pod kube-system/svclb-traefik-lcklq: failed to fetch token: serviceaccounts "default" is forbidden: User "system:node:raspberrypi" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.798932    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/projected/fa84d7db-7dd6-4247-9c0f-0ddba859b8d4-kube-api-access-fwzzz podName:fa84d7db-7dd6-4247-9c0f-0ddba859b8d4 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:19.298649929 +0000 GMT m=+12.957549984 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"kube-api-access-fwzzz\" (UniqueName: \"kubernetes.io/projected/fa84d7db-7dd6-4247-9c0f-0ddba859b8d4-kube-api-access-fwzzz\") pod \"svclb-traefik-lcklq\" (UID: \"fa84d7db-7dd6-4247-9c0f-0ddba859b8d4\") : failed to fetch token: serviceaccounts \"default\" is forbidden: User \"system:node:raspberrypi\" cannot create resource \"serviceaccounts/token\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'raspberrypi' and this object"
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.799083    8346 projected.go:199] Error preparing data for projected volume kube-api-access-vfncz for pod kube-system/coredns-7448499f4d-msgmv: failed to fetch token: serviceaccounts "coredns" is forbidden: User "system:node:raspberrypi" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kube-system": no relationship found between node 'raspberrypi' and this object
Nov 25 14:42:18 raspberrypi k3s[8346]: E1125 14:42:18.799403    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/projected/57269015-f305-405d-afa5-6ba6c8f645b5-kube-api-access-vfncz podName:57269015-f305-405d-afa5-6ba6c8f645b5 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:19.299330791 +0000 GMT m=+12.958230883 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"kube-api-access-vfncz\" (UniqueName: \"kubernetes.io/projected/57269015-f305-405d-afa5-6ba6c8f645b5-kube-api-access-vfncz\") pod \"coredns-7448499f4d-msgmv\" (UID: \"57269015-f305-405d-afa5-6ba6c8f645b5\") : failed to fetch token: serviceaccounts \"coredns\" is forbidden: User \"system:node:raspberrypi\" cannot create resource \"serviceaccounts/token\" in API group \"\" in the namespace \"kube-system\": no relationship found between node 'raspberrypi' and this object"
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.829266    8346 trace.go:205] Trace[1795834937]: "Create" url:/api/v1/nodes,user-agent:k3s/v1.21.6+k3s1 (linux/arm) kubernetes/df033fa,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (25-Nov-2021 14:42:15.718) (total time: 3110ms):
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1795834937]: [3.110476717s] [3.110476717s] END
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.848323    8346 trace.go:205] Trace[1005022848]: "Create" url:/api/v1/namespaces/default/events,user-agent:k3s/v1.21.6+k3s1 (linux/arm) kubernetes/df033fa,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (25-Nov-2021 14:42:15.671) (total time: 3165ms):
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1005022848]: ---"Object stored in database" 3164ms (14:42:00.836)
Nov 25 14:42:18 raspberrypi k3s[8346]: Trace[1005022848]: [3.165451328s] [3.165451328s] END
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.918713    8346 kubelet_node_status.go:109] "Node was previously registered" node="raspberrypi"
Nov 25 14:42:18 raspberrypi k3s[8346]: I1125 14:42:18.919500    8346 kubelet_node_status.go:74] "Successfully registered node" node="raspberrypi"
Nov 25 14:42:19 raspberrypi k3s[8346]: E1125 14:42:19.331264    8346 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
Nov 25 14:42:19 raspberrypi k3s[8346]: I1125 14:42:19.442429    8346 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
Nov 25 14:42:20 raspberrypi k3s[8346]: E1125 14:42:20.100684    8346 configmap.go:200] Couldn't get configMap kube-system/local-path-config: failed to sync configmap cache: timed out waiting for the condition
Nov 25 14:42:20 raspberrypi k3s[8346]: E1125 14:42:20.101436    8346 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/49ff49df-45fe-41f5-ad0e-3f1220f65d37-config-volume podName:49ff49df-45fe-41f5-ad0e-3f1220f65d37 nodeName:}" failed. No retries permitted until 2021-11-25 14:42:21.101343106 +0000 GMT m=+14.760243160 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49ff49df-45fe-41f5-ad0e-3f1220f65d37-config-volume\") pod \"local-path-provisioner-5ff76fc89d-z8gh9\" (UID: \"49ff49df-45fe-41f5-ad0e-3f1220f65d37\") : failed to sync configmap cache: timed out waiting for the condition"
Nov 25 14:42:20 raspberrypi k3s[8346]: I1125 14:42:20.289237    8346 prober_manager.go:255] "Failed to trigger a manual run" probe="Readiness"
Nov 25 14:42:21 raspberrypi k3s[8346]: I1125 14:42:21.672957    8346 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
Nov 25 14:42:21 raspberrypi k3s[8346]: I1125 14:42:21.673071    8346 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
Nov 25 14:42:21 raspberrypi k3s[8346]: I1125 14:42:21.770767    8346 scope.go:111] "RemoveContainer" containerID="081892bdffaf1e5ef168991de0c0d79aa3d104d9663ce136d8b2d692ff9312c6"
Nov 25 14:42:22 raspberrypi k3s[8346]: time="2021-11-25T14:42:22.134579252Z" level=info msg="labels have already set on node: raspberrypi"
Nov 25 14:42:22 raspberrypi k3s[8346]: time="2021-11-25T14:42:22.175817796Z" level=info msg="PodCIDR assigned for node raspberrypi"
Nov 25 14:42:22 raspberrypi k3s[8346]: I1125 14:42:22.198444    8346 flannel.go:93] Determining IP address of default interface
Nov 25 14:42:22 raspberrypi k3s[8346]: I1125 14:42:22.216029    8346 kube.go:120] Waiting 10m0s for node controller to sync
Nov 25 14:42:22 raspberrypi k3s[8346]: I1125 14:42:22.217671    8346 kube.go:378] Starting kube subnet manager
Nov 25 14:42:22 raspberrypi k3s[8346]: Flag --address has been deprecated, see --bind-address instead.
Nov 25 14:42:22 raspberrypi k3s[8346]: time="2021-11-25T14:42:22.236945926Z" level=info msg="Kube API server is now running"
Nov 25 14:42:22 raspberrypi k3s[8346]: time="2021-11-25T14:42:22.237095462Z" level=info msg="k3s is up and running"
Nov 25 14:42:22 raspberrypi k3s[8346]: I1125 14:42:22.238291    8346 controllermanager.go:175] Version: v1.21.6+k3s1
Nov 25 14:42:22 raspberrypi k3s[8346]: I1125 14:42:22.239635    8346 deprecated_insecure_serving.go:56] Serving insecurely on 127.0.0.1:10252
Nov 25 14:42:22 raspberrypi k3s[8346]: time="2021-11-25T14:42:22.240429365Z" level=info msg="Waiting for cloud-controller-manager privileges to become available"
Nov 25 14:42:22 raspberrypi k3s[8346]: W1125 14:42:22.518718    8346 authorization.go:47] Authorization is disabled
Nov 25 14:42:22 raspberrypi k3s[8346]: W1125 14:42:22.519730    8346 authentication.go:47] Authentication is disabled
Nov 25 14:42:22 raspberrypi k3s[8346]: I1125 14:42:22.520143    8346 deprecated_insecure_serving.go:54] Serving healthz insecurely on 127.0.0.1:10251
Nov 25 14:42:22 raspberrypi k3s[8346]: time="2021-11-25T14:42:22.716179723Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-9.18.2.tgz"
Nov 25 14:42:22 raspberrypi k3s[8346]: time="2021-11-25T14:42:22.717470744Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-9.18.2.tgz"
pi@raspberrypi:~ $ Nov 25 14:42:22 raspberrypi k3s[8346]: I1125 14:42:22.777399    8346 network_policy_controller.go:144] Starting network policy controller
Nov 25 14:42:22 raspberrypi systemd[1]: Started Lightweight Kubernetes.
Nov 25 14:42:22 raspberrypi k3s[8346]: W1125 14:42:22.976899    8346 container.go:586] Failed to update stats for container "/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/6d6f79c71c48c80d94294ee6b4bcc5aa74866a95a784b0d7966c54d0753c4638": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
Nov 25 14:42:22 raspberrypi k3s[8346]: failed to fetch hugetlb info
Nov 25 14:42:22 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
Nov 25 14:42:22 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
Nov 25 14:42:22 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
Nov 25 14:42:22 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
Nov 25 14:42:22 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
Nov 25 14:42:22 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
Nov 25 14:42:22 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd.(*containerdContainerHandler).GetStats
Nov 25 14:42:22 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/containerd/handler.go:188
Nov 25 14:42:22 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
Nov 25 14:42:22 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
Nov 25 14:42:22 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
Nov 25 14:42:22 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
Nov 25 14:42:22 raspberrypi k3s[8346]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
Nov 25 14:42:22 raspberrypi k3s[8346]:         /go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
Nov 25 14:42:22 raspberrypi k3s[8346]: runtime.goexit
Nov 25 14:42:22 raspberrypi k3s[8346]:         /usr/local/go/src/runtime/asm_arm.s:841], continuing to push stats
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.224148    8346 kube.go:127] Node controller sync successful
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.224323    8346 vxlan.go:137] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.234768    8346 kube.go:345] Skip setting NodeNetworkUnavailable
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.294008    8346 flannel.go:78] Wrote subnet file to /run/flannel/subnet.env
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.294067    8346 flannel.go:82] Running backend.
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.294092    8346 vxlan_network.go:60] watching for new subnet leases
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.341744190Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.342759603Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.343981718Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.344671598Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.347889799Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.349803386Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.373308836Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.375262275Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.376959957Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.379537555Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.381738898Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: time="2021-11-25T14:42:23.383310564Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.400758    8346 shared_informer.go:240] Waiting for caches to sync for tokens
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.717676    8346 shared_informer.go:247] Caches are synced for tokens
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.732629    8346 controllermanager.go:574] Started "ttl"
Nov 25 14:42:23 raspberrypi k3s[8346]: W1125 14:42:23.734509    8346 controllermanager.go:553] "tokencleaner" is disabled
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.736128    8346 ttl_controller.go:121] Starting TTL controller
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.736889    8346 shared_informer.go:240] Waiting for caches to sync for TTL
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.856002    8346 controllermanager.go:574] Started "job"
Nov 25 14:42:23 raspberrypi k3s[8346]: W1125 14:42:23.856081    8346 controllermanager.go:553] "cloud-node-lifecycle" is disabled
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.856690    8346 job_controller.go:150] Starting job controller
Nov 25 14:42:23 raspberrypi k3s[8346]: I1125 14:42:23.856734    8346 shared_informer.go:240] Waiting for caches to sync for job
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.074683    8346 controllermanager.go:574] Started "persistentvolume-expander"
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.075231    8346 expand_controller.go:327] Starting expand controller
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.076137    8346 shared_informer.go:240] Waiting for caches to sync for expand
Nov 25 14:42:24 raspberrypi k3s[8346]: time="2021-11-25T14:42:24.094684785Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
Nov 25 14:42:24 raspberrypi k3s[8346]: time="2021-11-25T14:42:24.099927276Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"be8b7fc9-281d-451f-bdaf-d70309c4480c\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"254\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.143356    8346 controllermanager.go:574] Started "ttl-after-finished"
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.144033    8346 ttlafterfinished_controller.go:109] Starting TTL after finished controller
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.144076    8346 shared_informer.go:240] Waiting for caches to sync for TTL after finished
Nov 25 14:42:24 raspberrypi k3s[8346]: time="2021-11-25T14:42:24.329276528Z" level=info msg="Starting /v1, Kind=Secret controller"
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.663117    8346 controllermanager.go:574] Started "garbagecollector"
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.683256    8346 garbagecollector.go:142] Starting garbage collector controller
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.683777    8346 shared_informer.go:240] Waiting for caches to sync for garbage collector
Nov 25 14:42:24 raspberrypi k3s[8346]: I1125 14:42:24.684391    8346 graph_builder.go:289] GraphBuilder running
Nov 25 14:42:24 raspberrypi k3s[8346]: time="2021-11-25T14:42:24.825229497Z" level=info msg="Cluster dns configmap already exists"
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.026106935Z" level=info msg="Starting /v1, Kind=Node controller"
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.026348395Z" level=info msg="Starting /v1, Kind=Service controller"
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.027439141Z" level=info msg="Starting /v1, Kind=Pod controller"
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.028066689Z" level=info msg="Starting /v1, Kind=Endpoints controller"
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.099747    8346 controllermanager.go:574] Started "daemonset"
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.100351    8346 daemon_controller.go:285] Starting daemon sets controller
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.100391    8346 shared_informer.go:240] Waiting for caches to sync for daemon sets
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.243336    8346 network_policy_controller.go:154] Starting network policy controller full sync goroutine
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.243681    8346 controllermanager.go:574] Started "replicationcontroller"
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.287817    8346 replica_set.go:182] Starting replicationcontroller controller
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.287913    8346 shared_informer.go:240] Waiting for caches to sync for ReplicationController
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.305320    8346 controller.go:611] quota admission added evaluator for: daemonsets.apps
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.307210845Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.307357899Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.308587014Z" level=info msg="Starting batch/v1, Kind=Job controller"
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.436547115Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"be8b7fc9-281d-451f-bdaf-d70309c4480c\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"254\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.518915    8346 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io
Nov 25 14:42:25 raspberrypi k3s[8346]: time="2021-11-25T14:42:25.664852475Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"b7a1fd8a-4cd1-4e45-b71a-370df306ec0c\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"268\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.746009    8346 controllermanager.go:574] Started "horizontalpodautoscaling"
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.746298    8346 horizontal.go:169] Starting HPA controller
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.808385    8346 shared_informer.go:240] Waiting for caches to sync for HPA
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.761783    8346 controller.go:611] quota admission added evaluator for: helmcharts.helm.cattle.io
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.853302    8346 controllermanager.go:574] Started "endpoint"
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.853844    8346 endpoints_controller.go:189] Starting endpoint controller
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.853879    8346 shared_informer.go:240] Waiting for caches to sync for endpoint
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.933733    8346 controllermanager.go:574] Started "endpointslice"
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.935576    8346 endpointslice_controller.go:256] Starting endpoint slice controller
Nov 25 14:42:25 raspberrypi k3s[8346]: I1125 14:42:25.935654    8346 shared_informer.go:240] Waiting for caches to sync for endpoint_slice
Nov 25 14:42:26 raspberrypi k3s[8346]: I1125 14:42:26.084263    8346 controllermanager.go:574] Started "disruption"
Nov 25 14:42:26 raspberrypi k3s[8346]: I1125 14:42:26.085189    8346 disruption.go:363] Starting disruption controller
Nov 25 14:42:26 raspberrypi k3s[8346]: I1125 14:42:26.085248    8346 shared_informer.go:240] Waiting for caches to sync for disruption
Nov 25 14:42:26 raspberrypi k3s[8346]: I1125 14:42:26.153210    8346 node_ipam_controller.go:91] Sending events to api server.
Nov 25 14:42:26 raspberrypi k3s[8346]: I1125 14:42:26.380871    8346 controller.go:611] quota admission added evaluator for: deployments.apps
Nov 25 14:42:26 raspberrypi k3s[8346]: time="2021-11-25T14:42:26.505642611Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"b7a1fd8a-4cd1-4e45-b71a-370df306ec0c\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"268\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
Nov 25 14:42:26 raspberrypi k3s[8346]: time="2021-11-25T14:42:26.583619166Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"local-storage\", UID:\"8171dabf-3b9c-4826-92c0-852bf824cc05\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"280\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\""
Nov 25 14:42:26 raspberrypi k3s[8346]: time="2021-11-25T14:42:26.990046140Z" level=info msg="Stopped tunnel to 127.0.0.1:6443"
Nov 25 14:42:26 raspberrypi k3s[8346]: time="2021-11-25T14:42:26.991165533Z" level=info msg="Connecting to proxy" url="wss://192.168.0.24:6443/v1-k3s/connect"
Nov 25 14:42:26 raspberrypi k3s[8346]: time="2021-11-25T14:42:26.991206773Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
Nov 25 14:42:26 raspberrypi k3s[8346]: time="2021-11-25T14:42:26.998810420Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.006750100Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"local-storage\", UID:\"8171dabf-3b9c-4826-92c0-852bf824cc05\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"280\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/local-storage.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.050443412Z" level=info msg="Handling backend connection request [raspberrypi]"
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.062824777Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"aggregated-metrics-reader\", UID:\"78c50c12-1d10-4203-8dd2-7fc11f710a60\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"285\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.103552033Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"aggregated-metrics-reader\", UID:\"78c50c12-1d10-4203-8dd2-7fc11f710a60\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"285\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.172558940Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-delegator\", UID:\"907a9149-62c9-4a49-a8ca-ebbff7371fd1\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"289\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.217900953Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-delegator\", UID:\"907a9149-62c9-4a49-a8ca-ebbff7371fd1\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"289\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.353215    8346 node.go:172] Successfully retrieved node IP: 192.168.0.24
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.353371    8346 server_others.go:141] Detected node IP 192.168.0.24
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.390050919Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-reader\", UID:\"c2b0e3e9-6e59-470c-a36d-bb31520cf976\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"294\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: E1125 14:42:27.461695    8346 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/322fa092c7576281456ac204667a5b943ae5f7c042c1e6b3a4776bc66f82c28d\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/f722a3f8c73e7c2b85bd09e7225a50cadd666948cda685eeee16278e5f4348ee\": RecentStats: unable to find data in memory cache], [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/9f6daa4c5ec9ef5a13cc7c7c29afd5798e4f7fdbe72772040f85c28af88d6a3a\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/4845d61da9079ed50f0d75a9d37bc340ead89a1f9605f1d771f7f0ef0da7467e\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/e66b33af61ae89800a904f8202a797fd3b2816f4285749718dc428cfa41544e0\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/c34ffa92938c3da7e450a153ba47c2be28cd418ebdfa10d5de6668ea7bd4d185\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podfa84d7db-7dd6-4247-9c0f-0ddba859b8d4/36abac6c6bb3005de20b48ba92c68735ef60ce6fc4e0e5b5697e267a4b640b43\": RecentStats: unable to find data in memory cache], [\"/kubepods\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/podeb4ae87c-ea92-4bdf-92d7-c05298ddb138/9027fc44f77ad6ee7c84464de008c071343d61d78fa2ecfc5879ae5052f6e109\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5/4724f2bebd6143ce5e5bb3d77495e807b700337cd5714e3136d328f6d501db45\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pod49ff49df-45fe-41f5-ad0e-3f1220f65d37/6d6f79c71c48c80d94294ee6b4bcc5aa74866a95a784b0d7966c54d0753c4638\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort/pode34c2e70-3667-4097-b5b0-052ffdac65d9/3c5a5d1c5d81fd0224de5d272b13f8dbc4c1b951fc5e5e03032bf2ccb5069b18\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod57269015-f305-405d-afa5-6ba6c8f645b5\": RecentStats: unable to find data in memory cache]"
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.554577034Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"auth-reader\", UID:\"c2b0e3e9-6e59-470c-a36d-bb31520cf976\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"294\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.611115094Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-apiservice\", UID:\"b08ca8d3-ba64-4b1e-a59a-dfec669f5534\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"301\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\""
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.701129    8346 server_others.go:207] kube-proxy running in dual-stack mode, IPv4-primary
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.701204    8346 server_others.go:213] Using iptables Proxier.
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.701236    8346 server_others.go:220] creating dualStackProxier for iptables.
Nov 25 14:42:27 raspberrypi k3s[8346]: W1125 14:42:27.701273    8346 server_others.go:502] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.780854    8346 server.go:647] Version: v1.21.6+k3s1
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.787915    8346 config.go:315] Starting service config controller
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.787969    8346 shared_informer.go:240] Waiting for caches to sync for service config
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.788024    8346 config.go:224] Starting endpoint slice config controller
Nov 25 14:42:27 raspberrypi k3s[8346]: I1125 14:42:27.788046    8346 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
Nov 25 14:42:27 raspberrypi k3s[8346]: time="2021-11-25T14:42:27.986403623Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-apiservice\", UID:\"b08ca8d3-ba64-4b1e-a59a-dfec669f5534\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"301\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml\""
Nov 25 14:42:28 raspberrypi k3s[8346]: I1125 14:42:28.037553    8346 shared_informer.go:247] Caches are synced for service config
Nov 25 14:42:28 raspberrypi k3s[8346]: I1125 14:42:28.094596    8346 shared_informer.go:247] Caches are synced for endpoint slice config
Nov 25 14:42:28 raspberrypi k3s[8346]: time="2021-11-25T14:42:28.189475356Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-deployment\", UID:\"f8a5a73f-13f0-46b4-a997-c897ec80f2f3\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"307\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\""
Nov 25 14:42:28 raspberrypi k3s[8346]: time="2021-11-25T14:42:28.348568725Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-deployment\", UID:\"f8a5a73f-13f0-46b4-a997-c897ec80f2f3\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"307\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml\""
Nov 25 14:42:28 raspberrypi k3s[8346]: time="2021-11-25T14:42:28.379207217Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-service\", UID:\"7e10d08a-9797-49a0-8a43-0b4b3f57e2f7\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"315\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\""
Nov 25 14:42:28 raspberrypi k3s[8346]: time="2021-11-25T14:42:28.617480405Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"metrics-server-service\", UID:\"7e10d08a-9797-49a0-8a43-0b4b3f57e2f7\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"315\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml\""
Nov 25 14:42:28 raspberrypi k3s[8346]: time="2021-11-25T14:42:28.752563440Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"resource-reader\", UID:\"d2088406-5e96-4786-8282-52ef0f541e6f\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"321\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\""
Nov 25 14:42:28 raspberrypi k3s[8346]: time="2021-11-25T14:42:28.968915361Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"resource-reader\", UID:\"d2088406-5e96-4786-8282-52ef0f541e6f\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"321\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml\""
Nov 25 14:42:29 raspberrypi k3s[8346]: time="2021-11-25T14:42:29.036926688Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"f19fbd88-8540-41e7-9563-a2a6d1283e33\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"329\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\""
Nov 25 14:42:29 raspberrypi k3s[8346]: time="2021-11-25T14:42:29.251652280Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rolebindings\", UID:\"f19fbd88-8540-41e7-9563-a2a6d1283e33\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"329\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/rolebindings.yaml\""
Nov 25 14:42:29 raspberrypi k3s[8346]: E1125 14:42:29.254647    8346 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubepods\": failed to get container info for \"/kubepods\": partial failures: [\"/kubepods\": RecentStats: unable to find data in memory cache]" containerName="/kubepods"
Nov 25 14:42:29 raspberrypi k3s[8346]: E1125 14:42:29.254778    8346 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/k3s.service\": failed to get container info for \"/system.slice/k3s.service\": partial failures: [\"/system.slice/k3s.service\": RecentStats: unable to find data in memory cache]" containerName="/system.slice/k3s.service"
Nov 25 14:42:29 raspberrypi k3s[8346]: E1125 14:42:29.254873    8346 helpers.go:673] "Eviction manager: failed to construct signal" err="system container \"pods\" not found in metrics" signal=allocatableMemory.available
Nov 25 14:42:29 raspberrypi k3s[8346]: time="2021-11-25T14:42:29.312527214Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"caeaf410-458c-4089-83ea-3576951f6672\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"337\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/traefik.yaml\""
Nov 25 14:42:29 raspberrypi k3s[8346]: time="2021-11-25T14:42:29.384834007Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"caeaf410-458c-4089-83ea-3576951f6672\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"337\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/traefik.yaml\""
Nov 25 14:42:33 raspberrypi k3s[8346]: I1125 14:42:33.099760    8346 serving.go:354] Generated self-signed cert in-memory
^C
pi@raspberrypi:~ $ Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.308549    8346 range_allocator.go:82] Sending events to api server.
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.309092    8346 range_allocator.go:110] No Service CIDR provided. Skipping filtering out service addresses.
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.309127    8346 range_allocator.go:116] No Secondary Service CIDR provided. Skipping filtering out secondary service addresses.
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.309241    8346 controllermanager.go:574] Started "nodeipam"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.309489    8346 node_ipam_controller.go:154] Starting ipam controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.309525    8346 shared_informer.go:240] Waiting for caches to sync for node
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.326010    8346 controllermanager.go:574] Started "podgc"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.326409    8346 gc_controller.go:89] Starting GC controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.326440    8346 shared_informer.go:240] Waiting for caches to sync for GC
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446027    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmchartconfigs.helm.cattle.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446155    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for helmcharts.helm.cattle.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446248    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for tlsstores.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446329    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for podtemplates
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446450    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446550    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for tlsoptions.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446692    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for middlewares.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: W1125 14:42:36.446743    8346 shared_informer.go:494] resyncPeriod 14h5m6.930672195s is smaller than resyncCheckPeriod 23h12m39.759579114s and the informer has already started. Changing it to 23h12m39.759579114s
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.446942    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for serviceaccounts
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447029    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for statefulsets.apps
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447089    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for deployments.apps
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447177    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for replicasets.apps
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447243    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for daemonsets.apps
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447349    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447452    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingressroutes.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447549    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingressrouteudps.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447627    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for serverstransports.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447696    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpoints
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447784    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for limitranges
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447903    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for events.events.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.447987    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448057    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for endpointslices.discovery.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448147    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for addons.k3s.cattle.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448254    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for controllerrevisions.apps
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448327    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448518    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for cronjobs.batch
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448593    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for jobs.batch
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448699    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingressroutetcps.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448796    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for traefikservices.traefik.containo.us
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448872    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.448960    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingresses.extensions
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.449041    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: W1125 14:42:36.449084    8346 shared_informer.go:494] resyncPeriod 22h13m56.82253773s is smaller than resyncCheckPeriod 23h12m39.759579114s and the informer has already started. Changing it to 23h12m39.759579114s
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.449369    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.449463    8346 resource_quota_monitor.go:229] QuotaMonitor created object count evaluator for csistoragecapacities.storage.k8s.io
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.449501    8346 controllermanager.go:574] Started "resourcequota"
Nov 25 14:42:36 raspberrypi k3s[8346]: W1125 14:42:36.449527    8346 controllermanager.go:553] "service" is disabled
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.450446    8346 resource_quota_controller.go:273] Starting resource quota controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.450485    8346 shared_informer.go:240] Waiting for caches to sync for resource quota
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.450527    8346 resource_quota_monitor.go:304] QuotaMonitor running
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.475460    8346 controllermanager.go:574] Started "attachdetach"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.476176    8346 attach_detach_controller.go:328] Starting attach detach controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.476220    8346 shared_informer.go:240] Waiting for caches to sync for attach detach
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.494481    8346 controllermanager.go:574] Started "root-ca-cert-publisher"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.494537    8346 publisher.go:102] Starting root CA certificate configmap publisher
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.494663    8346 shared_informer.go:240] Waiting for caches to sync for crt configmap
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.510987    8346 controllermanager.go:574] Started "ephemeral-volume"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.511012    8346 controller.go:170] Starting ephemeral volume controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.511120    8346 shared_informer.go:240] Waiting for caches to sync for ephemeral
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.527426    8346 controllermanager.go:574] Started "serviceaccount"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.527565    8346 serviceaccounts_controller.go:117] Starting service account controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.527596    8346 shared_informer.go:240] Waiting for caches to sync for service account
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.544362    8346 controllermanager.go:574] Started "csrapproving"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.544400    8346 certificate_controller.go:118] Starting certificate controller "csrapproving"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.545074    8346 shared_informer.go:240] Waiting for caches to sync for certificate-csrapproving
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.561722    8346 controllermanager.go:574] Started "statefulset"
Nov 25 14:42:36 raspberrypi k3s[8346]: W1125 14:42:36.562477    8346 controllermanager.go:553] "route" is disabled
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.561865    8346 stateful_set.go:146] Starting stateful set controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.563952    8346 shared_informer.go:240] Waiting for caches to sync for stateful set
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.580966    8346 controllermanager.go:574] Started "persistentvolume-binder"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.581375    8346 pv_controller_base.go:308] Starting persistent volume controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.581410    8346 shared_informer.go:240] Waiting for caches to sync for persistent volume
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.597592    8346 controllermanager.go:574] Started "endpointslicemirroring"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.598021    8346 endpointslicemirroring_controller.go:211] Starting EndpointSliceMirroring controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.598058    8346 shared_informer.go:240] Waiting for caches to sync for endpoint_slice_mirroring
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.614077    8346 controllermanager.go:574] Started "replicaset"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.614117    8346 replica_set.go:182] Starting replicaset controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.614714    8346 shared_informer.go:240] Waiting for caches to sync for ReplicaSet
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.634210    8346 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-serving"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.634258    8346 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-serving
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.634288    8346 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/server-ca.crt::/var/lib/rancher/k3s/server/tls/server-ca.key
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.636821    8346 certificate_controller.go:118] Starting certificate controller "csrsigning-kubelet-client"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.637403    8346 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kubelet-client
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.636891    8346 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.639442    8346 certificate_controller.go:118] Starting certificate controller "csrsigning-kube-apiserver-client"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.640035    8346 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-kube-apiserver-client
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.639537    8346 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.641808    8346 controllermanager.go:574] Started "csrsigning"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.642325    8346 certificate_controller.go:118] Starting certificate controller "csrsigning-legacy-unknown"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.642375    8346 shared_informer.go:240] Waiting for caches to sync for certificate-csrsigning-legacy-unknown
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.642475    8346 dynamic_serving_content.go:130] Starting csr-controller::/var/lib/rancher/k3s/server/tls/client-ca.crt::/var/lib/rancher/k3s/server/tls/client-ca.key
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.660368    8346 controllermanager.go:574] Started "csrcleaner"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.660787    8346 cleaner.go:82] Starting CSR cleaner controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.676642    8346 node_lifecycle_controller.go:377] Sending events to api server.
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.677342    8346 taint_manager.go:163] "Sending events to api server"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.677634    8346 node_lifecycle_controller.go:505] Controller will reconcile labels.
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.677723    8346 controllermanager.go:574] Started "nodelifecycle"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.678455    8346 node_lifecycle_controller.go:539] Starting node controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.678974    8346 shared_informer.go:240] Waiting for caches to sync for taint
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.693867    8346 controllermanager.go:574] Started "clusterrole-aggregation"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.693907    8346 clusterroleaggregation_controller.go:194] Starting ClusterRoleAggregator
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.694331    8346 shared_informer.go:240] Waiting for caches to sync for ClusterRoleAggregator
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.710297    8346 controllermanager.go:574] Started "pv-protection"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.710364    8346 pv_protection_controller.go:83] Starting PV protection controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.710395    8346 shared_informer.go:240] Waiting for caches to sync for PV protection
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.805023    8346 controllermanager.go:574] Started "namespace"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.805469    8346 namespace_controller.go:200] Starting namespace controller
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.805502    8346 shared_informer.go:240] Waiting for caches to sync for namespace
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.821702    8346 controllermanager.go:574] Started "cronjob"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.821734    8346 cronjob_controllerv2.go:125] Starting cronjob controller v2
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.822712    8346 shared_informer.go:240] Waiting for caches to sync for cronjob
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.840185    8346 controllermanager.go:574] Started "pvc-protection"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.840718    8346 pvc_protection_controller.go:110] "Starting PVC protection controller"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.840761    8346 shared_informer.go:240] Waiting for caches to sync for PVC protection
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.856755    8346 controllermanager.go:574] Started "deployment"
Nov 25 14:42:36 raspberrypi k3s[8346]: W1125 14:42:36.856805    8346 controllermanager.go:553] "bootstrapsigner" is disabled
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.857117    8346 deployment_controller.go:153] "Starting controller" controller="deployment"
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.857152    8346 shared_informer.go:240] Waiting for caches to sync for deployment
Nov 25 14:42:36 raspberrypi k3s[8346]: I1125 14:42:36.876197    8346 shared_informer.go:240] Waiting for caches to sync for resource quota
Nov 25 14:42:37 raspberrypi k3s[8346]: W1125 14:42:37.015012    8346 actual_state_of_world.go:534] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="raspberrypi" does not exist
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.017849    8346 shared_informer.go:247] Caches are synced for PV protection
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.022947    8346 shared_informer.go:247] Caches are synced for cronjob
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.028138    8346 shared_informer.go:247] Caches are synced for service account
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.037992    8346 shared_informer.go:247] Caches are synced for TTL
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.041632    8346 shared_informer.go:247] Caches are synced for endpoint_slice
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.051940    8346 shared_informer.go:247] Caches are synced for certificate-csrsigning-legacy-unknown
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.058182    8346 shared_informer.go:247] Caches are synced for deployment
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.063263    8346 shared_informer.go:247] Caches are synced for certificate-csrsigning-kube-apiserver-client
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.063906    8346 shared_informer.go:247] Caches are synced for PVC protection
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.064472    8346 shared_informer.go:247] Caches are synced for TTL after finished
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.065204    8346 shared_informer.go:247] Caches are synced for job
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.064612    8346 shared_informer.go:247] Caches are synced for stateful set
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.072435    8346 shared_informer.go:247] Caches are synced for endpoint
^C
pi@raspberrypi:~ $ ^C
pi@raspberrypi:~ $ Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.077583    8346 shared_informer.go:247] Caches are synced for expand
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.078314    8346 shared_informer.go:247] Caches are synced for attach detach
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.086510    8346 shared_informer.go:247] Caches are synced for persistent volume
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.096362    8346 shared_informer.go:247] Caches are synced for crt configmap
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.096794    8346 shared_informer.go:247] Caches are synced for ClusterRoleAggregator
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.099726    8346 shared_informer.go:247] Caches are synced for endpoint_slice_mirroring
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.109659    8346 shared_informer.go:247] Caches are synced for daemon sets
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.110551    8346 shared_informer.go:247] Caches are synced for namespace
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.111359    8346 shared_informer.go:247] Caches are synced for ephemeral
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.117285    8346 shared_informer.go:247] Caches are synced for node
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.117340    8346 range_allocator.go:172] Starting range CIDR allocator
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.117364    8346 shared_informer.go:240] Waiting for caches to sync for cidrallocator
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.117387    8346 shared_informer.go:247] Caches are synced for cidrallocator
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.118027    8346 shared_informer.go:247] Caches are synced for HPA
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.118701    8346 shared_informer.go:247] Caches are synced for ReplicaSet
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.130149    8346 shared_informer.go:247] Caches are synced for GC
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.138874    8346 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-client
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.139463    8346 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-serving
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.149515    8346 shared_informer.go:247] Caches are synced for certificate-csrapproving
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.250850    8346 shared_informer.go:247] Caches are synced for resource quota
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.280087    8346 shared_informer.go:247] Caches are synced for taint
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.280266    8346 node_lifecycle_controller.go:1398] Initializing eviction metric for zone:
Nov 25 14:42:37 raspberrypi k3s[8346]: W1125 14:42:37.280413    8346 node_lifecycle_controller.go:1013] Missing timestamp for Node raspberrypi. Assuming now as a timestamp.
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.280512    8346 node_lifecycle_controller.go:1214] Controller detected that zone  is now in state Normal.
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.281509    8346 shared_informer.go:247] Caches are synced for resource quota
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.282385    8346 taint_manager.go:187] "Starting NoExecuteTaintManager"
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.282917    8346 event.go:291] "Event occurred" object="raspberrypi" kind="Node" apiVersion="v1" type="Normal" reason="RegisteredNode" message="Node raspberrypi event: Registered Node raspberrypi in Controller"
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.292913    8346 shared_informer.go:247] Caches are synced for disruption
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.292958    8346 disruption.go:371] Sending events to api server.
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.297340    8346 shared_informer.go:247] Caches are synced for ReplicationController
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.434226    8346 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.435872    8346 shared_informer.go:240] Waiting for caches to sync for garbage collector
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.442320    8346 controller.go:611] quota admission added evaluator for: endpoints
Nov 25 14:42:37 raspberrypi k3s[8346]: W1125 14:42:37.630439    8346 authentication.go:308] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
Nov 25 14:42:37 raspberrypi k3s[8346]: W1125 14:42:37.630485    8346 authentication.go:332] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
Nov 25 14:42:37 raspberrypi k3s[8346]: W1125 14:42:37.630520    8346 authorization.go:184] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.638310    8346 shared_informer.go:247] Caches are synced for garbage collector
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.643397    8346 controllermanager.go:142] Version: v1.21.6+k3s1
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.646099    8346 secure_serving.go:202] Serving securely on 127.0.0.1:10258
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.646165    8346 tlsconfig.go:240] Starting DynamicServingCertificateController
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.684658    8346 shared_informer.go:247] Caches are synced for garbage collector
Nov 25 14:42:37 raspberrypi k3s[8346]: I1125 14:42:37.684742    8346 garbagecollector.go:151] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
vielmetti commented 2 years ago

You have lots of "hugetlb" mentions here, which may be related to #3517

vielmetti commented 2 years ago

and that issue references https://github.com/opencontainers/runc/issues/3232 which may be worth a second look

alexellis commented 2 years ago

Here's the k3s config check:

pi@raspberrypi:~ $ sudo k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/1e69816ee256b4bf0342f5188d55d937b9b7ada03be18500c0dd943b9ac0fff4/bin:
- sha256sum: good
- links: aux/ip6tables should link to iptables-detect.sh (fail)
- links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
- links: aux/ip6tables-save should link to iptables-detect.sh (fail)
- links: aux/iptables should link to iptables-detect.sh (fail)
- links: aux/iptables-restore should link to iptables-detect.sh (fail)
- links: aux/iptables-save should link to iptables-detect.sh (fail)

System:
- /var/lib/rancher/k3s/data/1e69816ee256b4bf0342f5188d55d937b9b7ada03be18500c0dd943b9ac0fff4/bin/aux iptables v1.8.6 (nf_tables): ok
- swap: should be disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: nonexistent?? (fail)
    (see https://github.com/tianon/cgroupfs-mount)
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled (as module)
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled (as module)
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 7 (fail)
pi@raspberrypi:~ $
alexellis commented 2 years ago

May be related to #3897

alexellis commented 2 years ago

@dweomer I'm happy to keep my RPi around until a fix is available here or upstream for the Raspberry Pi distro.

brandond commented 2 years ago

Looks like we'll need to update to runc 1.0.3 when it comes out: https://github.com/opencontainers/runc/pull/3295

brandond commented 2 years ago

Until then @alexellis you might see if the workaround from https://github.com/k3s-io/k3s/issues/4234#issuecomment-951993944 solves this for you?

joriskt commented 2 years ago

The referenced solution suggests to install a package only available for Ubuntu, but the original poster and I are both running Raspberry Pi OS (a Debian-based distro) in which said package does not exist.

The issue was runc, of which version 1.0.3 has since been released. When can we expect this to be available in a k3s release?

ShylajaDevadiga commented 2 years ago

Using commit id from master branch, having runc version 1.0.3 "failed to fetch hugetlb info" error from runc is not seen in logs

pi@raspberrypi:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

pi@raspberrypi:~ $ k3s -v
k3s version v1.22.4-rc3+k3s2 (5e73a828)
go version go1.16.10

pi@raspberrypi:~ $ runc --version
runc version 1.0.3
commit: v1.0.3-0-gf46b6ba2
spec: 1.0.2-dev
go: go1.16.10
libseccomp: 2.5.1

pi@raspberrypi:~ $ kubectl get nodes
NAME          STATUS   ROLES                  AGE     VERSION
raspberrypi   Ready    control-plane,master   9m51s   v1.22.4-rc3+k3s2

pi@raspberrypi:~ $ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-64ffb68fd-th2p2   1/1     Running     0          9m28s
kube-system   coredns-85cb69466-trcf2                  1/1     Running     0          9m28s
kube-system   metrics-server-9cf544f65-78q2l           1/1     Running     0          9m28s
kube-system   helm-install-traefik-crd--1-ksldk        0/1     Completed   0          9m28s
kube-system   helm-install-traefik--1-67d5z            0/1     Completed   2          9m28s
kube-system   svclb-traefik-glqgx                      2/2     Running     0          7m21s
kube-system   traefik-786ff64748-tfj9k                 1/1     Running     0          7m22s
pi@raspberrypi:~