Multiple containerd processes still running after stopping k0s

neopointer commented 5 days ago

Before creating an issue, make sure you've checked the following:

[x] You are running the latest released version of k0s
[x] Make sure you've searched for existing issues, both open and closed
[x] Make sure you've searched for PRs too, a fix might've been merged already
[x] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.37-1 (2023-07-03) x86_64 GNU/Linux
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Version

v1.31.2+k0s.0

Sysinfo

`k0s sysinfo`

Total memory: 23.5 GiB (pass)
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 1.1 TiB (pass)
Relative disk space available for /var/lib/k0s: 93% (pass)
Name resolution: localhost: [127.0.0.1 ::1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.1.0-10-amd64 (pass)
  Max. file descriptors per process: current: 524288 / max: 524288 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

After stopping k0s via k0s stop, I still see several process running.

root@prod01:~# ps aux | grep containerd
root         990  0.0  0.0 1238428 17168 ?       Sl   Nov22   2:40 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id c9ff81d8aa2a63ea7307304922dbc24647c12e4e76d4bcc770a8fb355a50e196 -address /run/k0s/containerd.sock
root         991  0.0  0.0 1238428 15972 ?       Sl   Nov22   2:30 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id d5436c809e10aee7b7a95643137d3e63eac7b04499133c31abdc032d8c7cafed -address /run/k0s/containerd.sock
root         992  0.0  0.0 1238172 15860 ?       Sl   Nov22   2:28 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 756ef3fe330f9cf9478b2b5d73774fc75729e79701f18b9282b4531e986e913b -address /run/k0s/containerd.sock
root        2510  0.0  0.0 1238172 16308 ?       Sl   Nov22   2:51 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id bfdbd88325ec9af79ee266b03e9f116c378e83335430e897d47bfbd45c216627 -address /run/k0s/containerd.sock
root        2511  0.0  0.0 1238236 15712 ?       Sl   Nov22   2:49 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id c0e570a657c235e38d8fdb6308e7c88d951c305e241b1fbcb1a2ab7b6efd7164 -address /run/k0s/containerd.sock
root        2586  0.0  0.0 1237916 15496 ?       Sl   Nov22   2:37 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b9dc16c1edbcc89f37a2dc99186b80f11bfd0351aa97496a63746987f899504 -address /run/k0s/containerd.sock
root        2599  0.1  0.0 1238428 15460 ?       Sl   Nov22   3:16 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id f4ecd67d95ce3d4c17780c9d233189a7efc6274befc04971c2829eb24c82ae19 -address /run/k0s/containerd.sock
root        3142  0.0  0.0 1238172 16100 ?       Sl   Nov22   2:54 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id cd6f3de43773a401a66c284e2ed7c46745dabcfaa5c060e0f677df5d5ea32ed5 -address /run/k0s/containerd.sock
root        3151  0.1  0.0 1238172 14868 ?       Sl   Nov22   3:42 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6aeca1681886e07b59c9031c311c3145bf920d5ee634ff20c1fd9f3377d051af -address /run/k0s/containerd.sock
root        3186  0.0  0.0 1238428 15220 ?       Sl   Nov22   2:46 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id f7018282aafc1edfd00cb8be75f13a0aab4f07860dad65cc09a51dc401815e57 -address /run/k0s/containerd.sock
root        3242  0.0  0.0 1238172 15536 ?       Sl   Nov22   2:30 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 132ba32fec02e9c2372373d010efb1528a5dbf3d51eaf9a805886f42b3b65f4c -address /run/k0s/containerd.sock
root        3631  0.0  0.0 1237916 15900 ?       Sl   Nov22   2:37 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5bf8e155c651fcfcc9183849db321d6a56773476feb311e61ad59e9e07290773 -address /run/k0s/containerd.sock
root        4168  0.0  0.0 1238172 15692 ?       Sl   Nov22   2:13 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 519d136b2abdc208102752723af708794d33ce3afc838b57688663c4b382817e -address /run/k0s/containerd.sock
root        4815  0.0  0.0 1238172 16212 ?       Sl   Nov22   2:39 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 936ad46fb454411fca0db60e7c274f70d8549c4fcbe1644840748a55627785f3 -address /run/k0s/containerd.sock
root      104830  0.1  0.0 1238172 15860 ?       Sl   12:24   0:11 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id 56943d2b26d1295243f3a0351fc48bef487bfed4d46b8a79aafae471beb38486 -address /run/k0s/containerd.sock
root      104864  0.1  0.0 1238428 15472 ?       Sl   12:24   0:10 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id a148a5e49e0ba7cd220f1b4e5f7bd686ee30fb9d978a10acab194a7bb935bcb2 -address /run/k0s/containerd.sock
root      108532  0.1  0.0 1238172 15804 ?       Sl   12:45   0:09 /var/lib/k0s/bin/containerd-shim-runc-v2 -namespace k8s.io -id e156c23799c0a410577360e9a589b5f92492627df5181bb1b7fd105636188581 -address /run/k0s/containerd.sock

I'm using Cilium, but I'm not sure if this has anything to do with it.

Steps to reproduce

I don't believe there are clear steps to reproduce, but try to use Cilium and see whether all processes stop.

Expected behavior

All processes related to k0s are gone after running k0s stop.

Actual behavior

Multiple containerd process are still running after using k0s stop.

Screenshots and logs

No response

Additional context

No response

twz123 commented 4 days ago

This is by design. k0s stop works similarly to kubelet / containerd. When these are stopped, containers continue to run. If you want to stop everything, you have to make sure that k0s doesn't start automatically after a reboot, and then restart the node. If you want to decommission the node, there's k0s reset, which goes some extra steps to try to terminate everything But: it will also wipe k0s's data directory.

neopointer commented 2 days ago

@twz123 in this case, this is not an issue, right? I would close it then.

Can we add this as documentation to k0s stop's documentation at least?

twz123 commented 13 hours ago

Can we add this as documentation to k0s stop's documentation at least?

Sure, what do you have in mind? PRs welcome, too 🙃

k0sproject / k0s