Closed UsernameAlvarez closed 4 months ago
I think some rpis linux distro is in /boot/firmware/nobtcmd.txt
I think some rpis linux distro is in
/boot/firmware/nobtcmd.txt
I added cgroup_enable=memory cgroup_memory=1 to/boot/firmware/nobtcmd.txt
as well, but it's didn't work.
ubuntu@ubuntu:/$ cat boot/firmware/nobtcmd.txt
cgroup_enable=memory cgroup_memory=1
microk8s inspect
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/1669/inspection-report-20201102_155946.tar.gz
@UsernameAlvarez just to confirm could you attach the inspection tarball?
I would like to know, this is on a real hardware, not any virtual machine on ARM64.
Hello @ktsakalozos is a raspberry pi 4 model B inspection-report-20201102_155946.tar.gz
In the API server logs I see:
Nov 02 15:01:25 ubuntu microk8s.daemon-apiserver[1993946]: Error: start node: raft_start(): io: load closed segment 0000000013312357-0000000013312881: found 524 entries (expected 525)
Nov 02 15:01:25 ubuntu systemd[1]: snap.microk8s.daemon-apiserver.service: Main process exited, code=exited, status=1/FAILURE
@freeekanayaka may have more insight on this error.
@UsernameAlvarez could you remove microk8s and reinstall it from the latest edge channel. We have a number of fixes in that channel and we will be releasing them soon.
sudo snap remove microk8s --purge
sudo snap install microk8s --classic --channel=1.19/edge
@ktsakalozos Thanks, I did it but is not working.
microk8s (1.19/edge) v1.19.3 from Canonical✓ installed
microk8s inspect
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/1783/inspection-report-20201104_230237.tar.gz
Inspection-report inspection-report-20201104_230237.tar.gz
@UsernameAlvarez can you cat the file /boot/firmware/cmdline.txt
. And cat /proc/cgroups
@balchua yes.
$ cat /boot/firmware/cmdline.txt
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc
cgroup_enable=memory cgroup_memory=1
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 5 1 1
cpu 8 1 1
cpuacct 8 1 1
blkio 3 1 1
memory 0 113 0
devices 9 104 1
freezer 10 2 1
net_cls 2 1 1
perf_event 6 1 1
net_prio 2 1 1
pids 4 111 1
rdma 7 1 1
I also have this problem, and have applied the 1.19/edge.
sudo snap install microk8s --classic --channel=1.19/edge
microk8s (1.19/edge) v1.19.4 from Canonical✓ installed
# Warning: iptables-legacy tables present, use iptables-legacy to see them
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
cat /boot/firmware/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 3 1 1
cpu 2 1 1
cpuacct 2 1 1
blkio 4 1 1
memory 0 117 0
devices 8 108 1
freezer 5 2 1
net_cls 9 1 1
perf_event 6 1 1
net_prio 9 1 1
pids 10 115 1
rdma 7 1 1
Same issue here with Ubuntu 20.04.2 on Raspberry Pi 4. Is any workaround to enable cgroup memory?
Same setup here (20.04.2 on Raspberry Pi 4), with latest 1.19
$ snap info microk8s | tail -1
installed: v1.19.9 (2098) 184MB classic
$ sudo lshw | grep Rasp
product: Raspberry Pi 4 Model B Rev 1.2
Adding
cgroup_enable=memory cgroup_memory=1
to /boot/firmware/cmdline.txt
and rebooting worked for me.
$ microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-control-plane-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting juju
Inspect Juju
Inspecting kubeflow
Inspect Kubeflow
Building the report tarball
Report tarball is at /var/snap/microk8s/2098/inspection-report-20210328_001211.tar.gz
I have enabled cgroups memory and microk8s does not want to see it.
# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 360 1
cpu 0 360 1
cpuacct 0 360 1
blkio 0 360 1
memory 0 360 1
devices 0 360 1
freezer 0 360 1
net_cls 0 360 1
perf_event 0 360 1
net_prio 0 360 1
hugetlb 0 360 1
pids 0 360 1
rdma 0 360 1
# microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-control-plane-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting juju
Inspect Juju
Inspecting kubeflow
Inspect Kubeflow
# Warning: iptables-legacy tables present, use iptables-legacy to see them
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/2213/inspection-report-20210601_104319.tar.gz
My OS is Debian Sid
Can confirm this issue with Debian 11, with microk8s 1.20.7 and 1.21.1
I have enabled cgroups memory and microk8s does not want to see it.
# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 0 360 1 cpu 0 360 1 cpuacct 0 360 1 blkio 0 360 1 memory 0 360 1 devices 0 360 1 freezer 0 360 1 net_cls 0 360 1 perf_event 0 360 1 net_prio 0 360 1 hugetlb 0 360 1 pids 0 360 1 rdma 0 360 1
My OS is Debian Sid
You probably didn't.
Did anyone tried adding cgroup_enable=memory cgroup_memory=1
into /etc/default/grub
to GRUB_CMDLINE_LINUX_DEFAULT
then reinstalling grub with update-grub
?
Also, try cat /proc/cmdline
to see if the option got there.
Yes, I tried. Output of microk8s inspect attached.
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.10.0-7-amd64 root=/dev/mapper/Rennschnecke-root ro cgroup_enable=memory cgroup_memory=1 quiet
$ microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-kubelite is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting juju
Inspect Juju
Inspecting kubeflow
Inspect Kubeflow
# Warning: iptables-legacy tables present, use iptables-legacy to see them
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/2210/inspection-report-20210622_082450.tar.gz
@mkesper i can see based on the inspect output, that the cluster seems to be in working condition.
Perhaps the microk8s output is the bug, which does not affect the cluster operation.
It seems so:
$ mount|grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
/src/upstream/microk8s (master) $ rg cgroup
...
scripts/inspect.sh
186: if ! mount | grep -q 'cgroup/memory'; then
187: printf -- '\033[0;33mWARNING: \033[0m The memory cgroup is not enabled. \n'
188: printf -- 'The cluster may not be functioning properly. Please ensure cgroups are enabled \n'
About cgroup error, my solution is add cgroup_enable=memory cgroup_memory=1
into the cmdline.txt of boot partition in SDCard, it works, and my operating system is rpios 64.
But the problem of iptables-legacy tables present, use iptables-legacy to see them
remains unresolved.
I've been trying to get microk8s 1.21/stable working on Ubuntu 21.10-arm64 on Pi-4 and it seems that perhaps microk8s doesn't like the new default in Ubuntu 21.10 ( from the release notes )
systemd is being switched to the “unified” cgroup hierarchy (cgroup v2) by default. If for some reason you need to keep the legacy cgroup v1 hierarchy, you can select it via a kernel parameter at boot time: systemd.unified_cgroup_hierarchy=0 (bug 1850667 29)
So - after adding systemd.unified_cgroup_hierarchy=0
the issue with the memory cgroup error, goes away.
(don't forget to rebuild the initramfs! sudo update-initramfs -u
)
To see which ( V1 or V2 ) you have configured -
stat -c %T -f /sys/fs/cgroup
V1 — tmpfs
V2 — cgroup2fs
If you get errors about iptables or iptables-legacy ( it seems that installing the microk8s snap installs these rules anyway ) these two commands will clear the tables.
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
sudo iptables-legacy -F && sudo iptables-legacy -t nat -F && sudo iptables-legacy -t mangle -F && sudo iptables-legacy -X
systemd, strikes again.
I've been trying to get microk8s 1.21/stable working on Ubuntu 21.10-arm64 on Pi-4 and it seems that perhaps microk8s doesn't like the new default in Ubuntu 21.10 ( from the release notes )
systemd is being switched to the “unified” cgroup hierarchy (cgroup v2) by default. If for some reason you need to keep the legacy cgroup v1 hierarchy, you can select it via a kernel parameter at boot time: systemd.unified_cgroup_hierarchy=0 (bug 1850667 29)
So - after adding
systemd.unified_cgroup_hierarchy=0
the issue with the memory cgroup error, goes away. (don't forget to rebuild the initramfs!sudo update-initramfs -u
) ...systemd, strikes again.
This is not a long time solution but a workaround for the fact that microk8s (at least 1.21 which isn't the most up to date anymore) does not support cgroup v2 properly and uses iptables-legacy for its whole network configuration. NB: The cluster worked for me apparently with cgroup v2 but there was this irritating message.
Problem present on my system as well. Looks like the inspect test is faulty.
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 206 1
cpu 0 206 1
cpuacct 0 206 1
blkio 0 206 1
memory 0 206 1
devices 0 206 1
freezer 0 206 1
net_cls 0 206 1
perf_event 0 206 1
net_prio 0 206 1
hugetlb 0 206 1
pids 0 206 1
rdma 0 206 1
misc 0 206 1
microk8s inspect
<snip>
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-41-generic root=UUID=80...16 ro zswap.enabled=1 zswap.compressor=zstd cgroup_enable=memory cgroup_memory=1 cgroup_enable=cpuset
snap list | grep microk8s
microk8s v1.21.12 3202 1.21/stable canonical** classic
uname -a
Linux muon 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Hello, I'm running Ubuntu 22.04.1 on arm64 (m1 in parallels) and in my case /boot/cmdline.txt or /boot/firmware/cmdline.txt simply don't exist. Any ideas where I'm supposed to be adding the settings related to cgroups? I'm encoutering the exact same issue when trying to use microk8s 1.19. @hkmaly I have tried enabling the cgroup memory by adding it to the kernel parameters in grub, but it doesn't make any difference.
I did switch to cgroup v1, but take a really long time to reinstall microk8s or to get information about the node (initially), so I'm rather sceptical of this solution.
in my case /boot/cmdline.txt or /boot/firmware/cmdline.txt simply don't exist. Any ideas where I'm supposed to be adding the settings related to cgroups?
I have tried enabling the cgroup memory by adding it to the kernel parameters in grub, but it doesn't make any difference.
There. I mean, those are kernel parameters, the cmdline.txt is just way how to get it to grub. If you are not sure what kernel parameters actually got into running kernel, look into /proc/cmdline
.
(Note: I gave up on kubernetes for now and hope that when I will have time for them again it will be in better shape.)
@hkmaly you can always use a higher version though. It works without any issues. This is a special case.
@lethargosapatheia ... higher version of what? You mean like microk9s? :smile:
Yes.
Here is a fix for this in ubuntu 22.04. Open the file /etc/default/grub
in an editor. Find where the string GRUB_CMDLINE_LINUX
is set. Add cgroup_enable=memory cgroup_memory=1
to that string. Save the file and exit the editor. Then run sudo update-grub
. Here is how my grub file looks like:
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX="cgroup_enable=memory cgroup_memory=1 swapaccount=1 systemd.unified_cgroup_hierarchy=0"
Then you may restart your computer. Then run microk8s.inspect
, everything should be ok. Below is my microk8s version
MicroK8s v1.25.2 revision 4055
GRUB_CMDLINE_LINUX="cgroup_enable=memory cgroup_memory=1 systemd.unified_cgroup_hierarchy=0"
I needed all three of these.
same issue installed on Ubuntu core 22.04 LTS - Raspberry pi
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
mount |grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
microk8s version
MicroK8s v1.25.2 revision 4054
sudo microk8s inspect
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/4054/inspection-report-20221023_225317.tar.gz
had the same problem on 22.10 on amd64
Same here, amd64 working fine before Ubuntu upgrade, should I try to change cgroups to v1?
This issue has been been fixed by https://github.com/canonical/microk8s/pull/3518.
For anyone installing MicroK8S on Ubuntu 22.04 not raspberry pi here is a link to the steps I took: Microk8s setup on ubuntu 22.04
What about debian? Or devuan?
Have you experienced the same issue on them? Enabling Cgroups should generally work the same way for server-based OSs. Once you have added the following and updated your grub and initramfs. You should be good to go.
GRUB_CMDLINE_LINUX="cgroup_enable=memory cgroup_memory=1 systemd.unified_cgroup_hierarchy=0"
What about debian? Or devuan?
I had the same issue in Raspberry Pi 4 B even after adding cgroup_enable=memory cgroup_memory=1
.
Then I installed docker and reboot and tried to start again there was an error showing:
WARNING: Docker is installed. File "/etc/docker/daemon.json" does not exist. You should create it and add the following lines:
{
"insecure-registries" : ["localhost:32000"]
}
and then restart docker with: sudo systemctl restart docker.
I did that and sudo reboot
and the error is gone
.
Not sure what is the relationship, but you may try also and feeback!
Hi @armanfatahi, the warning printed and the suggested fix is for the local deployment of docker to be able to push to the local registry you can enable with microk8s enable registry
. I do not see how this could be related to the cgroup issue.
I have a similar issue:
I have microk8s cluster in raspberry pi 4 with rasbian
node1: master (and also worker) with kernel: 5.15.76-v8+ node2: worker only with kernel: 5.15.84-v8+
If I add a memory limit to a deployment and if it starts in worker node I get this error (only if memory limit is on)
Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: openat2 /sys/fs/cgroup/kubepods/burstable/pod13a4b4e3-c07f-4178-8b8c-1d801660d60d/fetcher/memory.max: no such file or directory: unknown
It it starts on the master node the pod starts successfully.
[microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
FAIL: Service snap.microk8s.daemon-k8s-dqlite is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-k8s-dqlite
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-traefik is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting juju
Inspect Juju
Inspecting kubeflow
Inspect Kubeflow
Inspecting dqlite
Inspect dqlite
Building the report tarball
Report tarball is at /var/snap/microk8s/4600/inspection-report-20230215_004055.tar.gz]
I just ran into this oddity on 22.04, microk8s 1.25. Claims that memory cgroup is not enabled when it clearly is. I've applied all the suggested "fixes" and it didn't change anything. I don't think there's anything wrong with my cluster, but that the check is broken.
I just ran into this oddity on 22.04, microk8s 1.25. Claims that memory cgroup is not enabled when it clearly is. I've applied all the suggested "fixes" and it didn't change anything. I don't think there's anything wrong with my cluster, but that the check is broken.
am having the same issue too, and I am on 1.27 and Ubuntu 22.04.....why does this keep happening? cgroups2 is most definitely enabled
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
I appended
cgroup_enable=memory cgroup_memory=1
tocmdline.txt
file in each node and reboot but it not working.microk8s inspect