Kubernetes doesn't wanto to start

[sync-by-unito[bot]]

Hi,

I’m in trouble with multiple case on Truenas scale doens’t want to start properly after a boot.

The big problem is related to kubernetes that doesn’t want to start. Looking at it’s status I see always a “KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized” and the inizializing phase remain in append even for days without a reason.

The network works properly, the GUI is accessable (I got logs and I can manage Truenas) but I don’t know why kubernetes doens’t go on.

Another point is that during boot phase I noticed 2 services in error:

ix-swap.service

ix-etc.service

!immagine-20240613-093541.png|width=1064,height=667,alt="immagine-20240613-093541.png"!

Wnat happens?

What can I do to resolve it?

I’m using truenas scale in a lot of envirnment and this kind of problems are blastering me. I cannot go on with it.

┆Attachments: immagine-20240613-093541.png

[sync-by-unito[bot]]

➤ Rosario Pagano commented:

Sorry, I forgot some required info. I use Truenas scale 23.10.2 with a dedicated pool for ixapp.

from k3s describe

Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message

MemoryPressure False Thu, 13 Jun 2024 11:35:07 0200 Thu, 13 Jun 2024 11:14:22 0200 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 13 Jun 2024 11:35:07 0200 Thu, 13 Jun 2024 11:14:22 0200 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 13 Jun 2024 11:35:07 0200 Thu, 13 Jun 2024 11:14:22 0200 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Thu, 13 Jun 2024 11:35:07 0200 Thu, 13 Jun 2024 11:14:22 0200 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized Addresses: InternalIP: x.x.x.x Hostname: ix-truenas

with this log you can see network issue advice

[sync-by-unito[bot]]

➤ Rosario Pagano commented:

Hi guys,

any news? I cannot start kubernetes and all related containers.

Can someone help?

Please rememeber that that happens every time I boot TrueNAS. In some cases after restarting ix-etc and reset kubernetes pool it goes in running but if I reboot the system the problem came up again.

No way to do a complete boot and have all servers running.

Regards

Rosario

[sync-by-unito[bot]]

➤ Rosario Pagano commented:

Hi,

same other news about this issue.

Looking at logs on system I noticed a lot of coredumps related to systemd-journal daemon.

what happens?

May this help to understand the root couse?

Thankd in advance

Rosario

drwxr-xr-x 2 root root 36 Jun 17 09:50 . drwxr-xr-x 9 root root 10 Mar 15 11:53 .. rwr----- 1 root root 291639 Jun 16 03:15 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1059937.1718500523000000.zst rwr----- 1 root root 292364 Jun 16 05:49 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1096007.1718509790000000.zst rwr----- 1 root root 292644 Jun 16 07:05 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1131410.1718514309000000.zst rwr----- 1 root root 266537 Jun 16 07:42 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1154817.1718516576000000.zst rwr----- 1 root root 13 Jun 16 08:58 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1179542.1718521093000000.zst rwr----- 1 root root 261544 Jun 16 09:23 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1190468.1718522629000000.zst rwr----- 1 root root 264790 Jun 16 11:15 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1224841.1718529319000000.zst rwr----- 1 root root 265298 Jun 16 12:07 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1247335.1718532441000000.zst rwr----- 1 root root 291884 Jun 16 13:24 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1250191.1718537096000000.zst rwr----- 1 root root 257727 Jun 16 14:46 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1297497.1718541991000000.zst rwr----- 1 root root 288237 Jun 16 16:03 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1300356.1718546598000000.zst rwr----- 1 root root 308525 Jun 16 18:24 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1335892.1718555049000000.zst rwr----- 1 root root 275615 Jun 16 19:40 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1382817.1718559653000000.zst rwr----- 1 root root 255408 Jun 16 20:00 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1395191.1718560833000000.zst rwr----- 1 root root 278759 Jun 17 00:22 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1475553.1718576536000000.zst rwr----- 1 root root 256157 Jun 17 02:59 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1524136.1718585960000000.zst rwr----- 1 root root 276688 Jun 17 03:38 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1534672.1718588335000000.zst rwr----- 1 root root 289899 Jun 17 04:11 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1546970.1718590319000000.zst rwr----- 1 root root 289650 Jun 17 05:22 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1571047.1718594568000000.zst rwr----- 1 root root 285376 Jun 17 05:52 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1580043.1718596370000000.zst rwr----- 1 root root 307984 Jun 17 09:50 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.1614948.1718610629000000.zst rwr----- 1 root root 259435 Jun 14 08:39 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.266701.1718347180000000.zst rwr----- 1 root root 264922 Jun 14 10:09 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.280570.1718352574000000.zst rwr----- 1 root root 272347 Jun 14 11:42 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.308599.1718358121000000.zst rwr----- 1 root root 277591 Jun 14 12:28 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.337539.1718360884000000.zst rwr----- 1 root root 297859 Jun 15 04:39 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.352039.1718419197000000.zst rwr----- 1 root root 302008 Jun 14 07:56 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.364.1718344559000000.zst rwr----- 1 root root 287049 Jun 15 07:44 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.656710.1718430243000000.zst rwr----- 1 root root 223657 Jun 15 08:28 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.714319.1718432890000000.zst rwr----- 1 root root 299233 Jun 15 18:21 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.871791.1718468515000000.zst rwr----- 1 root root 13 Jun 15 20:29 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.915310.1718476172000000.zst rwr----- 1 root root 283455 Jun 15 21:49 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.967352.1718480999000000.zst rwr----- 1 root root 265999 Jun 15 22:35 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.980585.1718483724000000.zst rwr----- 1 root root 265408 Jun 15 23:59 core.systemd-journal.0.49ad5ecf8b204e5182a72befe4fce656.994631.1718488791000000.zst

[sync-by-unito[bot]]

➤ Muhammad Rehan commented:

Rosario Pagano can you please update to dragonfish and see if this is still a problem ? Secondly please upload a debug of your system when it is in an erroneous state..thanks!