WillardHu
commented
6 months ago It may be that edgemesh's iptables rules are confused, if you can reinstall edgemesh, you can try it.
Open 120L020430 opened 6 months ago
WillardHu
commented
6 months ago It may be that edgemesh's iptables rules are confused, if you can reinstall edgemesh, you can try it.
120L020430
commented
6 months ago Ok. I will try it. Thanks.
120L020430
commented
6 months ago It may be that edgemesh's iptables rules are confused, if you can reinstall edgemesh, you can try it.
Thank you again. I uninstalled edgemesh, cleared iptables rules and reinstalled edgemesh. But the problem happened again. A log of edgecore caught my eye, which is as follows.
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
I changed my default runtime of containerd from nvidia to runc. Luckily, it worked.
Later, I tried to use nvidia-smi and got Failed to initialize NVML: Driver/library version mismatch. So my driver of gpu is wrong. But I remember that several days ago, I could get right results of nvidia-smi, while the problem of this issue existed. So I still don't know the crash problem is caused by whether the wrong driver or wrong configuration of containerd runtime.
I will keep finding it out.
federicocagnola
commented
6 months ago I seem to have encountered the same problem, but I have a feeling it is not related to edgemesh as my deployment does not include it.
Instead i noticed this issue today, while trying to bump Ubuntu on the edge node from 20.04 to 22.04. I never had any such issue with Ubuntu20 LTS, but started having the exact same behavior when i tried the newer Ubuntu on the edge node. I see @120L020430 also has Ubuntu22 on the edge node.
I have tried deploying several pods, a redis instance, an mqtt instance and a custom python image, and they all experience the same problem. Code works fine, logs look great, then the container receives a shutdown signal and stops running processes.
At the same time edgecore logs that it failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff, as if the process within the container was never successfully started. I wonder if this could be related to the OS, or with the way kubeedge can tell whether a container was successfully started or not.
Mar 08 14:42:07 edgeNode edgecore[53223]: E0308 14:42:07.273208 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"core\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=core pod=core-edgeNode-685fb7b7fc-mjscv_edge-namespace(ac734606-ea10-4c75-bf4a-b7b92674c62d)\"" pod="edge-namespace/core-edgeNode-685fb7b7fc-mjscv" podUID="ac734606-ea10-4c75-bf4a-b7b92674c62d"
Mar 08 14:42:21 edgeNode edgecore[53223]: I0308 14:42:21.259117 53223 scope.go:115] "RemoveContainer" containerID="eb7c16e8b4df87b4f2e068b2358b41e651648e7e4174194fea46c58812f1c7db"
Mar 08 14:42:50 edgeNode edgecore[53223]: I0308 14:42:50.656521 53223 scope.go:115] "RemoveContainer" containerID="ed43fab20a36292da72be8ff08dbb56ba9f095344dd63d382e625356fe39cbcd"
Mar 08 14:42:50 edgeNode edgecore[53223]: I0308 14:42:50.656655 53223 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c28857cfa8e86ec4c4bfc28cf0329618a23924f8eac7e649480dd1def8cf0b"
Mar 08 14:42:51 edgeNode edgecore[53223]: E0308 14:42:51.054033 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:42:51 edgeNode edgecore[53223]: I0308 14:42:51.802070 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:42:51 edgeNode edgecore[53223]: E0308 14:42:51.802629 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:05 edgeNode edgecore[53223]: I0308 14:43:05.265078 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:05 edgeNode edgecore[53223]: E0308 14:43:05.265541 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:17 edgeNode edgecore[53223]: I0308 14:43:17.277374 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:17 edgeNode edgecore[53223]: E0308 14:43:17.277870 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:32 edgeNode edgecore[53223]: I0308 14:43:32.407199 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:32 edgeNode edgecore[53223]: E0308 14:43:32.407680 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:44 edgeNode edgecore[53223]: I0308 14:43:44.728456 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:44 edgeNode edgecore[53223]: E0308 14:43:44.728938 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:56 edgeNode edgecore[53223]: I0308 14:43:56.272156 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:56 edgeNode edgecore[53223]: E0308 14:43:56.272574 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"kubectl view of the node:
kubectl get pods -A -owide | grep edgeNode
edge-namespace core-edgeNode-685fb7b7fc-mjscv 1/1 Running 6 (2m46s ago) 32m 10.1.8.7 edgeNode <none> <none>
edge-namespace mqtt-local-edgeNode-5cd99f69d-zs7jj 1/1 Running 8 (109s ago) 33m 10.1.8.7 edgeNode <none> <none>
edge-namespace redis-edgeNode-65dd65d6c7-lkjdh 0/1 CrashLoopBackOff 7 (19s ago) 32m 10.1.8.7 edgeNode <none> <none>I managed to get kubeedge working on Ubuntu22 @120L020430 @WillardHu After thorough investigation it looked like the culprit was cgroups v2 and container compatibility. I hope that following my steps will allow other developers to solve this issue for everyone
References:
To fix cgroup management to work with kubeedge:
sudo nano /etc/default/grubi edited the following line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"to this:
GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0 quiet splash"then
sudo update-gruband reboot
What happened: There is a node called knode5 in my kubeedge cluster. After restarting service edgecore in the node, the pods in it can run correctly in a little while (no exact time). But then all pods will irregularly switch back and forth between CrashLoopBackOff and Running state like the below.
What you expected to happen: All the pods in the node can run correctly at least for a long time. How to reproduce it (as minimally and precisely as possible): Sorry, this problem happened about three weeks ago. Before then, there was nothing wrong with the pods in the node. Just on the next day of fixing the
nvidia device plugin, the node was suddenly trapped in the problem. I tried to delete nvidia device plugin, but the problem still existed. So I think the problem has nothing with the plugin and I don't know how to reproduce it.Anything else we need to know?: Some information on the node:
pod
nginxlog:pod
lclog:pod
edgemeshlog:latest 40 logs of
edgecore:I tried to fix the error
"Failed to get the info of the filesystem with mountpoint" err="cannot find filesystem info for device \"/dev/nvme0n1p5\"" mountpoint="/etc/containerd/containerd/io.containerd.snapshotter.v1.overlayfs". I changed the storage location of ‘contained’, but no matter whether it’s on different partitions of the solid-state drive or the mechanical hard drive, this error will be reported. I don't know whether the error is important. Environment:kubectl version): Client Version: v1.26.7, Kustomize Version: v4.5.7, Server Version: v1.26.11cloudcore --versionandedgecore --version): v1.15.0Cloud nodes Environment:
- Hardware configuration (e.g. `lscpu`): - ``` 架构： x86_64 CPU 运行模式： 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual 字节序： Little Endian CPU: 6 在线 CPU 列表： 0-5 厂商 ID： GenuineIntel 型号名称： Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz ``` - OS (e.g. `cat /etc/os-release`): Ubuntu 22.04.3 LTS - Kernel (e.g. `uname -a`): Linux master 6.5.0-14-generic \#14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux - Go version (e.g. `go version`): go version go1.19.12 linux/amd64 - Others:Edge nodes Environment:
- edgecore version (e.g. `edgecore --version`): v1.15.0 - Hardware configuration (e.g. `lscpu`): - ``` 架构： x86_64 CPU 运行模式： 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual 字节序： Little Endian CPU: 6 在线 CPU 列表： 0-5 厂商 ID： GenuineIntel 型号名称： Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz ``` - OS (e.g. `cat /etc/os-release`): Ubuntu 22.04.3 LTS - Kernel (e.g. `uname -a`): Linux knode5 6.5.0-15-generic \#15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux - Go version (e.g. `go version`): go version go1.18.1 linux/amd64 - Others: