Open 120L020430 opened 6 months ago
It may be that edgemesh's iptables rules are confused, if you can reinstall edgemesh, you can try it.
Ok. I will try it. Thanks.
It may be that edgemesh's iptables rules are confused, if you can reinstall edgemesh, you can try it.
Thank you again. I uninstalled edgemesh, cleared iptables rules and reinstalled edgemesh. But the problem happened again. A log of edgecore caught my eye, which is as follows.
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown
I changed my default runtime of containerd from nvidia to runc. Luckily, it worked.
Later, I tried to use nvidia-smi
and got Failed to initialize NVML: Driver/library version mismatch
. So my driver of gpu is wrong. But I remember that several days ago, I could get right results of nvidia-smi
, while the problem of this issue existed. So I still don't know the crash problem is caused by whether the wrong driver or wrong configuration of containerd runtime.
I will keep finding it out.
I seem to have encountered the same problem, but I have a feeling it is not related to edgemesh as my deployment does not include it.
Instead i noticed this issue today, while trying to bump Ubuntu on the edge node from 20.04 to 22.04. I never had any such issue with Ubuntu20 LTS, but started having the exact same behavior when i tried the newer Ubuntu on the edge node. I see @120L020430 also has Ubuntu22 on the edge node.
I have tried deploying several pods, a redis
instance, an mqtt
instance and a custom python
image, and they all experience the same problem. Code works fine, logs look great, then the container receives a shutdown signal and stops running processes.
At the same time edgecore logs that it failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff
, as if the process within the container was never successfully started. I wonder if this could be related to the OS, or with the way kubeedge can tell whether a container was successfully started or not.
Mar 08 14:42:07 edgeNode edgecore[53223]: E0308 14:42:07.273208 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"core\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=core pod=core-edgeNode-685fb7b7fc-mjscv_edge-namespace(ac734606-ea10-4c75-bf4a-b7b92674c62d)\"" pod="edge-namespace/core-edgeNode-685fb7b7fc-mjscv" podUID="ac734606-ea10-4c75-bf4a-b7b92674c62d"
Mar 08 14:42:21 edgeNode edgecore[53223]: I0308 14:42:21.259117 53223 scope.go:115] "RemoveContainer" containerID="eb7c16e8b4df87b4f2e068b2358b41e651648e7e4174194fea46c58812f1c7db"
Mar 08 14:42:50 edgeNode edgecore[53223]: I0308 14:42:50.656521 53223 scope.go:115] "RemoveContainer" containerID="ed43fab20a36292da72be8ff08dbb56ba9f095344dd63d382e625356fe39cbcd"
Mar 08 14:42:50 edgeNode edgecore[53223]: I0308 14:42:50.656655 53223 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c28857cfa8e86ec4c4bfc28cf0329618a23924f8eac7e649480dd1def8cf0b"
Mar 08 14:42:51 edgeNode edgecore[53223]: E0308 14:42:51.054033 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:42:51 edgeNode edgecore[53223]: I0308 14:42:51.802070 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:42:51 edgeNode edgecore[53223]: E0308 14:42:51.802629 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:05 edgeNode edgecore[53223]: I0308 14:43:05.265078 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:05 edgeNode edgecore[53223]: E0308 14:43:05.265541 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:17 edgeNode edgecore[53223]: I0308 14:43:17.277374 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:17 edgeNode edgecore[53223]: E0308 14:43:17.277870 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:32 edgeNode edgecore[53223]: I0308 14:43:32.407199 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:32 edgeNode edgecore[53223]: E0308 14:43:32.407680 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:44 edgeNode edgecore[53223]: I0308 14:43:44.728456 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:44 edgeNode edgecore[53223]: E0308 14:43:44.728938 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
Mar 08 14:43:56 edgeNode edgecore[53223]: I0308 14:43:56.272156 53223 scope.go:115] "RemoveContainer" containerID="eb23b5df38ccf5f4dced0454f79c2bfb3bc3f39bdd7a43166d395c687ae54c3f"
Mar 08 14:43:56 edgeNode edgecore[53223]: E0308 14:43:56.272574 53223 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"redis\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=redis pod=redis-edgeNode-65dd65d6c7-lkjdh_edge-namespace(525c9cf6-ed98-464a-b4c5-bda0dee8a1e6)\"" pod="edge-namespace/redis-edgeNode-65dd65d6c7-lkjdh" podUID="525c9cf6-ed98-464a-b4c5-bda0dee8a1e6"
kubectl view of the node:
kubectl get pods -A -owide | grep edgeNode
edge-namespace core-edgeNode-685fb7b7fc-mjscv 1/1 Running 6 (2m46s ago) 32m 10.1.8.7 edgeNode <none> <none>
edge-namespace mqtt-local-edgeNode-5cd99f69d-zs7jj 1/1 Running 8 (109s ago) 33m 10.1.8.7 edgeNode <none> <none>
edge-namespace redis-edgeNode-65dd65d6c7-lkjdh 0/1 CrashLoopBackOff 7 (19s ago) 32m 10.1.8.7 edgeNode <none> <none>
I managed to get kubeedge working on Ubuntu22 @120L020430 @WillardHu After thorough investigation it looked like the culprit was cgroups v2 and container compatibility. I hope that following my steps will allow other developers to solve this issue for everyone
References:
To fix cgroup management to work with kubeedge:
sudo nano /etc/default/grub
i edited the following line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
to this:
GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0 quiet splash"
then
sudo update-grub
and reboot
What happened: There is a node called knode5 in my kubeedge cluster. After restarting service edgecore in the node, the pods in it can run correctly in a little while (no exact time). But then all pods will irregularly switch back and forth between CrashLoopBackOff and Running state like the below.
What you expected to happen: All the pods in the node can run correctly at least for a long time. How to reproduce it (as minimally and precisely as possible): Sorry, this problem happened about three weeks ago. Before then, there was nothing wrong with the pods in the node. Just on the next day of fixing the
nvidia device plugin
, the node was suddenly trapped in the problem. I tried to delete nvidia device plugin, but the problem still existed. So I think the problem has nothing with the plugin and I don't know how to reproduce it.Anything else we need to know?: Some information on the node:
pod
nginx
log:pod
lc
log:pod
edgemesh
log:latest 40 logs of
edgecore
:I tried to fix the error
"Failed to get the info of the filesystem with mountpoint" err="cannot find filesystem info for device \"/dev/nvme0n1p5\"" mountpoint="/etc/containerd/containerd/io.containerd.snapshotter.v1.overlayfs"
. I changed the storage location of ‘contained’, but no matter whether it’s on different partitions of the solid-state drive or the mechanical hard drive, this error will be reported. I don't know whether the error is important. Environment:kubectl version
): Client Version: v1.26.7, Kustomize Version: v4.5.7, Server Version: v1.26.11cloudcore --version
andedgecore --version
): v1.15.0Cloud nodes Environment:
- Hardware configuration (e.g. `lscpu`): - ``` 架构: x86_64 CPU 运行模式: 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual 字节序: Little Endian CPU: 6 在线 CPU 列表: 0-5 厂商 ID: GenuineIntel 型号名称: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz ``` - OS (e.g. `cat /etc/os-release`): Ubuntu 22.04.3 LTS - Kernel (e.g. `uname -a`): Linux master 6.5.0-14-generic \#14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux - Go version (e.g. `go version`): go version go1.19.12 linux/amd64 - Others:Edge nodes Environment:
- edgecore version (e.g. `edgecore --version`): v1.15.0 - Hardware configuration (e.g. `lscpu`): - ``` 架构: x86_64 CPU 运行模式: 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual 字节序: Little Endian CPU: 6 在线 CPU 列表: 0-5 厂商 ID: GenuineIntel 型号名称: Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz ``` - OS (e.g. `cat /etc/os-release`): Ubuntu 22.04.3 LTS - Kernel (e.g. `uname -a`): Linux knode5 6.5.0-15-generic \#15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux - Go version (e.g. `go version`): go version go1.18.1 linux/amd64 - Others: