Can't create multinode cluster with podman (tried with docker that worked) - 1 control-plane and 1 worker

[aadi58002]

What happened: Tried to create multi node cluster with podman fails

What you expected to happen: For two containers with 1 control-plane and 1 worker node to be created

How to reproduce it (as minimally and precisely as possible): sudo kind create cluster --config multicontrol-plane-cluster.yml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker 

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.20.0 go1.21.1 linux/amd64
• Runtime info: (use docker info or podman info):

  host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
  package: Unknown
  path: /usr/libexec/podman/conmon
  version: 'conmon version 2.1.7, commit: unknown'
  cpuUtilization:
  idlePercent: 95.61
  systemPercent: 2.19
  userPercent: 2.2
  cpus: 16
  databaseBackend: boltdb
  distribution:
  distribution: '"void"'
  version: unknown
  eventLogger: file
  hostname: void-live
  idMappings:
  gidmap: null
  uidmap: null
  kernel: 6.5.9_1
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 9766612992
  memTotal: 16160567296
  networkBackend: cni
  ociRuntime:
  name: runc
  package: Unknown
  path: /usr/bin/runc
  version: |-
    runc version 1.1.9
    spec: 1.0.2-dev
    go: go1.21.0
    libseccomp: 2.5.4
  os: linux
  remoteSocket:
  exists: true
  path: /run/podman/podman.sock
  security:
  apparmorEnabled: false
  capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
  rootless: false
  seccompEnabled: true
  seccompProfilePath: /usr/share/containers/seccomp.json
  selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
  executable: /usr/bin/slirp4netns
  package: Unknown
  version: |-
    slirp4netns version 1.2.0
    commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
    libslirp: 4.7.0
    SLIRP_CONFIG_VERSION_MAX: 4
    libseccomp: 2.5.4
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 0h 7m 54.00s
  plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
  registries: {}
  store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
  number: 2
  paused: 0
  running: 2
  stopped: 0
  graphDriverName: overlay
  graphOptions:
  overlay.mount_program:
    Executable: /usr/bin/fuse-overlayfs
    Package: Unknown
    Version: |-
      fusermount3 version: 3.16.1
      fuse-overlayfs: version 1.11
      FUSE library version 3.16.1
      using FUSE kernel interface version 7.38
  overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 699229667328
  graphRootUsed: 10089136128
  graphStatus:
  Backing Filesystem: zfs
  Native Overlay Diff: "false"
  Supports d_type: "true"
  Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
  number: 2
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
  version:
  APIVersion: 4.5.1
  Built: 0
  BuiltTime: Thu Jan  1 05:30:00 1970
  GitCommit: ""
  GoVersion: go1.21.0
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

• OS (e.g. from /etc/os-release):

  
  NAME="Void"
  ID="void"
  PRETTY_NAME="Void Linux"
  HOME_URL="https://voidlinux.org/"
  DOCUMENTATION_URL="https://docs.voidlinux.org/"
  LOGO="void-logo"
  ANSI_COLOR="0;38;2;71;128;97"

DISTRIB_ID="void"

- Kubernetes version: (use `kubectl version`): Client Version: v1.28.2
- Any proxies or other special environment settings?: 
I am on a non systemd distro and i am not using network manage ( if it is significant in anyway )

Projects/Image-build-files/kind on  main [✘!?] took 11s ❯ sudo kind create cluster --config multicontrol-plane-cluster.yml enabling experimental podman provider Creating cluster "kind" ... ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 ✓ Preparing nodes 📦 📦
✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✗ Joining worker nodes 🚜 Deleted nodes: ["kind-worker" "kind-control-plane"] ERROR: failed to create cluster: failed to join node with kubeadm: command "podman exec --privileged kind-worker kubeadm join --config /kind/kubeadm.conf --skip-phases=preflight --v=6" failed with error: exit status 1 Command Output: I1101 16:18:30.485288 158 join.go:412] [preflight] found NodeName empty; using OS hostname as NodeName I1101 16:18:30.485502 158 joinconfiguration.go:76] loading configuration from "/kind/kubeadm.conf" I1101 16:18:30.486263 158 controlplaneprepare.go:225] [download-certs] Skipping certs download I1101 16:18:30.486275 158 join.go:529] [preflight] Discovering cluster-info I1101 16:18:30.486284 158 token.go:80] [discovery] Created cluster-info discovery client, requesting info from "kind-control-plane:6443" I1101 16:18:30.626360 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 137 milliseconds I1101 16:18:30.626446 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:18:37.173664 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 178 milliseconds I1101 16:18:37.173767 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:18:43.116418 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 287 milliseconds I1101 16:18:43.116506 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:18:49.485886 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 72 milliseconds I1101 16:18:49.485979 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:18:55.708295 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 259 milliseconds I1101 16:18:55.708378 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:01.852388 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 180 milliseconds I1101 16:19:01.852490 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:07.484757 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 279 milliseconds I1101 16:19:07.484840 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:13.730588 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 342 milliseconds I1101 16:19:13.730703 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:19.180286 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 184 milliseconds I1101 16:19:19.180392 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:25.399801 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 102 milliseconds I1101 16:19:25.399881 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:31.752956 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 420 milliseconds I1101 16:19:31.753043 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:37.180059 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 262 milliseconds I1101 16:19:37.180152 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:42.402047 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 141 milliseconds I1101 16:19:42.402144 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:48.247594 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 303 milliseconds I1101 16:19:48.247672 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:19:53.973271 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 345 milliseconds I1101 16:19:53.973326 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:00.014969 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 344 milliseconds I1101 16:20:00.015057 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:05.851414 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 156 milliseconds I1101 16:20:05.851505 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:11.586712 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 342 milliseconds I1101 16:20:11.586795 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:18.068108 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 72 milliseconds I1101 16:20:18.068208 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:24.087296 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 348 milliseconds I1101 16:20:24.087408 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:29.508495 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 70 milliseconds I1101 16:20:29.508601 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:35.756364 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 67 milliseconds I1101 16:20:35.756454 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:42.307131 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 316 milliseconds I1101 16:20:42.307228 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:48.860892 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 310 milliseconds I1101 16:20:48.860970 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:20:54.278846 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 63 milliseconds I1101 16:20:54.278930 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:00.269840 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 106 milliseconds I1101 16:21:00.269927 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:06.382788 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 106 milliseconds I1101 16:21:06.382868 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:12.389023 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 58 milliseconds I1101 16:21:12.389120 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:18.223899 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 63 milliseconds I1101 16:21:18.223994 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:23.779022 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 176 milliseconds I1101 16:21:23.779104 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:28.890813 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 61 milliseconds I1101 16:21:28.890906 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:34.069994 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 102 milliseconds I1101 16:21:34.070092 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:39.343995 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 156 milliseconds I1101 16:21:39.344104 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:45.451246 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 67 milliseconds I1101 16:21:45.451363 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:51.325695 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 324 milliseconds I1101 16:21:51.325849 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:21:57.879063 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 336 milliseconds I1101 16:21:57.879186 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:04.535037 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 327 milliseconds I1101 16:22:04.535121 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:10.947543 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 82 milliseconds I1101 16:22:10.947648 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:16.239395 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 72 milliseconds I1101 16:22:16.239456 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:21.830952 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 79 milliseconds I1101 16:22:21.831054 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:27.852983 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 69 milliseconds I1101 16:22:27.853069 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:34.026982 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 281 milliseconds I1101 16:22:34.027066 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:40.170490 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 205 milliseconds I1101 16:22:40.170597 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:45.326959 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 72 milliseconds I1101 16:22:45.327045 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:50.840769 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 141 milliseconds I1101 16:22:50.840848 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:22:57.477500 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 360 milliseconds I1101 16:22:57.477617 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:23:03.781336 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 103 milliseconds I1101 16:23:03.781417 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:23:09.969533 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 200 milliseconds I1101 16:23:09.969591 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:23:15.159641 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 98 milliseconds I1101 16:23:15.159725 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:23:20.620203 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 218 milliseconds I1101 16:23:20.620290 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host I1101 16:23:25.945774 158 round_trippers.go:553] GET https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s in 280 milliseconds I1101 16:23:25.945881 158 token.go:217] [discovery] Failed to request cluster-info, will try again: Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host Get "https://kind-control-plane:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp: lookup kind-control-plane on 1.1.1.1:53: no such host couldn't validate the identity of the API Server k8s.io/kubernetes/cmd/kubeadm/app/discovery.For cmd/kubeadm/app/discovery/discovery.go:45 k8s.io/kubernetes/cmd/kubeadm/app/cmd.(joinData).TLSBootstrapCfg cmd/kubeadm/app/cmd/join.go:530 k8s.io/kubernetes/cmd/kubeadm/app/cmd.(joinData).InitCfg cmd/kubeadm/app/cmd/join.go:540 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.getKubeletStartJoinData cmd/kubeadm/app/cmd/phases/join/kubelet.go:91 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletStartJoinPhase cmd/kubeadm/app/cmd/phases/join/kubelet.go:106 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run.func1 cmd/kubeadm/app/cmd/phases/workflow/runner.go:259 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).visitAll cmd/kubeadm/app/cmd/phases/workflow/runner.go:446 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run cmd/kubeadm/app/cmd/phases/workflow/runner.go:232 k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1 cmd/kubeadm/app/cmd/join.go:179 github.com/spf13/cobra.(Command).execute vendor/github.com/spf13/cobra/command.go:916 github.com/spf13/cobra.(Command).ExecuteC vendor/github.com/spf13/cobra/command.go:1040 github.com/spf13/cobra.(Command).Execute vendor/github.com/spf13/cobra/command.go:968 k8s.io/kubernetes/cmd/kubeadm/app.Run cmd/kubeadm/app/kubeadm.go:50 main.main cmd/kubeadm/kubeadm.go:25 runtime.main /usr/local/go/src/runtime/proc.go:250 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598 error execution phase kubelet-start k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run.func1 cmd/kubeadm/app/cmd/phases/workflow/runner.go:260 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).visitAll cmd/kubeadm/app/cmd/phases/workflow/runner.go:446 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run cmd/kubeadm/app/cmd/phases/workflow/runner.go:232 k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdJoin.func1 cmd/kubeadm/app/cmd/join.go:179 github.com/spf13/cobra.(Command).execute vendor/github.com/spf13/cobra/command.go:916 github.com/spf13/cobra.(Command).ExecuteC vendor/github.com/spf13/cobra/command.go:1040 github.com/spf13/cobra.(Command).Execute vendor/github.com/spf13/cobra/command.go:968 k8s.io/kubernetes/cmd/kubeadm/app.Run cmd/kubeadm/app/kubeadm.go:50 main.main cmd/kubeadm/kubeadm.go:25 runtime.main /usr/local/go/src/runtime/proc.go:250 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598

[send.tgz](https://github.com/kubernetes-sigs/kind/files/13229302/send.tgz)

[aadi58002]

I tried to setup multi node cluster on a friends laptop which does not use zfs and it worked there.

Also in the logs of the containers it mentions that it fails to load some modules.

~/.config on  main [!?] ❯ sudo podman exec -it kind-worker bash
root@kind-worker:/# systemctl status systemd-modules-load.service
● systemd-modules-load.service - Load Kernel Modules
     Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static)
     Active: failed (Result: exit-code) since Fri 2023-11-24 15:26:46 UTC; 37min ago
       Docs: man:systemd-modules-load.service(8)
             man:modules-load.d(5)
    Process: 99 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
   Main PID: 99 (code=exited, status=1/FAILURE)
        CPU: 17ms

Warning: journal has been rotated since unit was started, output may be incomplete.
root@kind-worker:/# cat /lib/systemd/system/systemd-modules-load.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Load Kernel Modules
Documentation=man:systemd-modules-load.service(8) man:modules-load.d(5)
DefaultDependencies=no
Conflicts=shutdown.target
Before=sysinit.target shutdown.target
ConditionCapability=CAP_SYS_MODULE
ConditionDirectoryNotEmpty=|/lib/modules-load.d
ConditionDirectoryNotEmpty=|/usr/lib/modules-load.d
ConditionDirectoryNotEmpty=|/usr/local/lib/modules-load.d
ConditionDirectoryNotEmpty=|/etc/modules-load.d
ConditionDirectoryNotEmpty=|/run/modules-load.d
ConditionKernelCommandLine=|modules-load
ConditionKernelCommandLine=|rd.modules-load

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/lib/systemd/systemd-modules-load
TimeoutSec=90s
root@kind-worker:/# systemctl restart systemd-modules-load.service
Job for systemd-modules-load.service failed because the control process exited with error code.
See "systemctl status systemd-modules-load.service" and "journalctl -xe" for details.
root@kind-worker:/# journalctl -xe
-- Journal begins at Fri 2023-11-24 15:26:46 UTC, ends at Fri 2023-11-24 16:05:50 UTC. --
Nov 24 15:26:46 kind-worker systemd-journald[98]: Journal started
░░ Subject: The journal has been started
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The system journal process has started up, opened the journal
░░ files for writing and is now ready to process requests.
Nov 24 15:26:46 kind-worker systemd-journald[98]: Runtime Journal (/run/log/journal/c77f1b43487c4fb087d4596a0cc81790) is 8.0M, max 770.5M, 762.5M free.
░░ Subject: Disk space used by the journal
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Runtime Journal (/run/log/journal/c77f1b43487c4fb087d4596a0cc81790) is currently using 8.0M.
░░ Maximum allowed usage is set to 770.5M.
░░ Leaving at least 385.3M free (of currently available 7.5G of disk space).
░░ Enforced usage limit is thus 770.5M, of which 762.5M are still available.
░░
░░ The limits controlling how much disk space is used by the journal may
░░ be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
░░ RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
░░ /etc/systemd/journald.conf. See journald.conf(5) for details.
Nov 24 15:26:46 kind-worker systemd-modules-load[99]: Failed to insert module 'iscsi_tcp': Exec format error
Nov 24 15:26:46 kind-worker systemd-modules-load[99]: Failed to insert module 'ib_iser': Exec format error
Nov 24 15:26:46 kind-worker systemd-sysusers[102]: Creating group systemd-timesync with gid 999.
Nov 24 15:26:46 kind-worker systemd-sysusers[102]: Creating user systemd-timesync (systemd Time Synchronization) with uid 999 and gid 999.
Nov 24 15:26:46 kind-worker systemd-sysusers[102]: Creating group systemd-coredump with gid 998.
Nov 24 15:26:46 kind-worker systemd-sysusers[102]: Creating user systemd-coredump (systemd Core Dumper) with uid 998 and gid 998.
Nov 24 15:26:46 kind-worker systemd[1]: Starting Flush Journal to Persistent Storage...
░░ Subject: A start job for unit systemd-journal-flush.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-journal-flush.service has begun execution.
░░
░░ The job identifier is 28.
Nov 24 15:26:46 kind-worker systemd[1]: Finished Apply Kernel Variables.
░░ Subject: A start job for unit systemd-sysctl.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-sysctl.service has finished successfully.
░░
░░ The job identifier is 27.
Nov 24 15:26:46 kind-worker systemd[1]: Finished Create System Users.
░░ Subject: A start job for unit systemd-sysusers.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-sysusers.service has finished successfully.
░░
░░ The job identifier is 12.
Nov 24 15:26:46 kind-worker systemd-journald[98]: Runtime Journal (/run/log/journal/c77f1b43487c4fb087d4596a0cc81790) is 8.0M, max 770.5M, 762.5M free.
░░ Subject: Disk space used by the journal
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Runtime Journal (/run/log/journal/c77f1b43487c4fb087d4596a0cc81790) is currently using 8.0M.
░░ Maximum allowed usage is set to 770.5M.
░░ Leaving at least 385.3M free (of currently available 7.5G of disk space).
░░ Enforced usage limit is thus 770.5M, of which 762.5M are still available.
░░
░░ The limits controlling how much disk space is used by the journal may
░░ be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
░░ RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
░░ /etc/systemd/journald.conf. See journald.conf(5) for details.
Nov 24 15:26:46 kind-worker systemd[1]: Starting Create Static Device Nodes in /dev...
░░ Subject: A start job for unit systemd-tmpfiles-setup-dev.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-tmpfiles-setup-dev.service has begun execution.
░░
░░ The job identifier is 14.
Nov 24 15:26:46 kind-worker systemd[1]: Finished Flush Journal to Persistent Storage.
░░ Subject: A start job for unit systemd-journal-flush.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-journal-flush.service has finished successfully.
░░
░░ The job identifier is 28.
Nov 24 15:26:46 kind-worker systemd[1]: Finished Create Static Device Nodes in /dev.
░░ Subject: A start job for unit systemd-tmpfiles-setup-dev.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-tmpfiles-setup-dev.service has finished successfully.
░░
░░ The job identifier is 14.
Nov 24 15:26:46 kind-worker systemd[1]: Reached target Local File Systems (Pre).
░░ Subject: A start job for unit local-fs-pre.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit local-fs-pre.target has finished successfully.
░░
░░ The job identifier is 38.
Nov 24 15:26:46 kind-worker systemd[1]: Reached target Local File Systems.
░░ Subject: A start job for unit local-fs.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit local-fs.target has finished successfully.
░░
░░ The job identifier is 36.
Nov 24 15:26:46 kind-worker systemd[1]: Condition check resulted in Store a System Token in an EFI Variable being skipped.
░░ Subject: A start job for unit systemd-boot-system-token.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-boot-system-token.service has finished successfully.
░░
░░ The job identifier is 10.
Nov 24 15:26:46 kind-worker systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
░░ Subject: A start job for unit systemd-machine-id-commit.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-machine-id-commit.service has finished successfully.
░░
░░ The job identifier is 34.
Nov 24 15:26:46 kind-worker systemd[1]: Condition check resulted in Rule-based Manager for Device Events and Files being skipped.
░░ Subject: A start job for unit systemd-udevd.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-udevd.service has finished successfully.
░░
░░ The job identifier is 5.
Nov 24 15:26:46 kind-worker systemd[1]: Reached target System Initialization.
░░ Subject: A start job for unit sysinit.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit sysinit.target has finished successfully.
░░
░░ The job identifier is 4.
Nov 24 15:26:46 kind-worker systemd[1]: Started Daily Cleanup of Temporary Directories.
░░ Subject: A start job for unit systemd-tmpfiles-clean.timer has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-tmpfiles-clean.timer has finished successfully.
░░
░░ The job identifier is 52.
Nov 24 15:26:46 kind-worker systemd[1]: Reached target Basic System.
░░ Subject: A start job for unit basic.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit basic.target has finished successfully.
░░
░░ The job identifier is 3.
Nov 24 15:26:46 kind-worker systemd[1]: Reached target Timers.
░░ Subject: A start job for unit timers.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit timers.target has finished successfully.
░░
░░ The job identifier is 51.
Nov 24 15:26:46 kind-worker systemd[1]: Starting containerd fuse-overlayfs snapshotter...
░░ Subject: A start job for unit containerd-fuse-overlayfs.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd-fuse-overlayfs.service has begun execution.
░░
░░ The job identifier is 54.
Nov 24 15:26:46 kind-worker systemd[1]: Starting Undo KIND mount hacks...
░░ Subject: A start job for unit undo-mount-hacks.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit undo-mount-hacks.service has begun execution.
░░
░░ The job identifier is 56.
Nov 24 15:26:46 kind-worker systemd[1]: undo-mount-hacks.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit undo-mount-hacks.service has successfully entered the 'dead' state.
Nov 24 15:26:46 kind-worker systemd[1]: Finished Undo KIND mount hacks.
░░ Subject: A start job for unit undo-mount-hacks.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit undo-mount-hacks.service has finished successfully.
░░
░░ The job identifier is 56.
Nov 24 15:26:46 kind-worker containerd-fuse-overlayfs-grpc[107]: time="2023-11-24T15:26:46Z" level=info msg="containerd-fuse-overlayfs-grpc Version=\"v1.0.5\" Revision=\"11c45f4d24689d8cb279813fbcb9bbd01773e0e8\""
Nov 24 15:26:46 kind-worker systemd[1]: Starting containerd container runtime...
░░ Subject: A start job for unit containerd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has begun execution.
░░
░░ The job identifier is 55.
Nov 24 15:26:46 kind-worker systemd[1]: Condition check resulted in kubelet: The Kubernetes Node Agent being skipped.
░░ Subject: A start job for unit kubelet.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit kubelet.service has finished successfully.
░░
░░ The job identifier is 57.
Nov 24 15:26:46 kind-worker systemd[1]: var-lib-containerd\x2dfuse\x2doverlayfs-fuseoverlayfs\x2dcheck3138333460-merged.mount: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit var-lib-containerd\x2dfuse\x2doverlayfs-fuseoverlayfs\x2dcheck3138333460-merged.mount has successfully entered the 'dead' state.
Nov 24 15:26:46 kind-worker modprobe[113]: modprobe: ERROR: could not insert 'overlay': Exec format error
Nov 24 15:26:46 kind-worker systemd[1]: Started containerd fuse-overlayfs snapshotter.
░░ Subject: A start job for unit containerd-fuse-overlayfs.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd-fuse-overlayfs.service has finished successfully.
░░
░░ The job identifier is 54.
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.074503615Z" level=info msg="starting containerd" revision=1677a17964311325ed1c31e2c0a3589ce6d5c30d version=v1.7.1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.118201324Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.118310348Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.118366711Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.118776968Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.fuse-overlayfs\"..." type=io.containerd.snapshotter.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.120510746Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.120571788Z" level=info msg="metadata content store policy set" policy=shared
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126175835Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126262580Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126295965Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126351699Z" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126384036Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126411974Z" level=info msg="NRI interface is disabled by configuration."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126436838Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126567863Z" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126612702Z" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126643223Z" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126672697Z" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126704126Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126733739Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126763353Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126792407Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126822440Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126874962Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126905483Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.126931953Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.127011434Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.128147008Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.128266440Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.128301221Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.128350880Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129059714Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129098687Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129128230Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129157774Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129212251Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129251782Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129280487Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129309332Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.129345860Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.130893437Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.130943095Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.130973128Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131002671Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131030748Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131063714Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131092279Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131117982Z" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131653816Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:fuse-overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox} test-handler:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:true IgnoreBlockIONotEnabledErrors:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginSetupSerially:false NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.k8s.io/pause:3.7 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false EnableCDI:false CDISpecDirs:[/etc/cdi /var/run/cdi] ImagePullProgressTimeout:1m0s DrainExecSyncIOTimeout:0s} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131800346Z" level=info msg="Connect containerd service"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131856569Z" level=info msg="using legacy CRI server"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.131875776Z" level=info msg="using experimental NRI integration - disable nri plugin to prevent this"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.132055831Z" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.fuse-overlayfs\""
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.134301975Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.136588559Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.136639195Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.136671672Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.136694231Z" level=info msg="Start subscribing containerd event"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.136708130Z" level=info msg="skipping tracing processor initialization (no tracing plugin)" error="no OpenTelemetry endpoint: skip plugin"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.136862482Z" level=info msg="Start recovering state"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.137393567Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.137515373Z" level=info msg=serving... address=/run/containerd/containerd.sock
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.137637947Z" level=info msg="containerd successfully booted in 0.065027s"
Nov 24 15:26:47 kind-worker systemd[1]: Started containerd container runtime.
░░ Subject: A start job for unit containerd.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit containerd.service has finished successfully.
░░
░░ The job identifier is 55.
Nov 24 15:26:47 kind-worker systemd[1]: Reached target Multi-User System.
░░ Subject: A start job for unit multi-user.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit multi-user.target has finished successfully.
░░
░░ The job identifier is 2.
Nov 24 15:26:47 kind-worker systemd[1]: Reached target Graphical Interface.
░░ Subject: A start job for unit graphical.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit graphical.target has finished successfully.
░░
░░ The job identifier is 1.
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.142080089Z" level=warning msg="The image registry.k8s.io/coredns/coredns:v1.10.1 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.143006344Z" level=warning msg="The image sha256:b0b1fa0f58c6e932b7f20bf208b2841317a1e8c88cc51b18358310bbd8ec95da is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.143245695Z" level=warning msg="The image registry.k8s.io/kube-proxy:v1.27.3 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.143260642Z" level=warning msg="The image sha256:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.144064323Z" level=warning msg="The image sha256:221177c6082a88ea4f6240ab2450d540955ac6f4d5454f0e15751b653ebda165 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.144498326Z" level=warning msg="The image registry.k8s.io/pause:3.7 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.145367380Z" level=warning msg="The image registry.k8s.io/kube-apiserver:v1.27.3 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.145578655Z" level=warning msg="The image import-2023-06-15@sha256:ce2145a147b3f1fc440ba15eaa91b879ba9cbf929c8dd8f3190868f4373f2183 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.145673571Z" level=warning msg="The image registry.k8s.io/kube-scheduler:v1.27.3 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.145783154Z" level=warning msg="The image import-2023-06-15@sha256:bdbeb95d8a0820cbc385e44f75ed25799ac8961e952ded26aa2a09b3377dfee7 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.145845105Z" level=warning msg="The image registry.k8s.io/etcd:3.5.7-0 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.145868432Z" level=warning msg="The image sha256:9d5429f6d7697ae3186f049e142875ba5854f674dfee916fa6c53da276808a23 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.146889674Z" level=warning msg="The image docker.io/kindest/local-path-provisioner:v20230511-dc714da8 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.146991644Z" level=warning msg="The image registry.k8s.io/kube-controller-manager:v1.27.3 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.146922151Z" level=warning msg="The image sha256:be300acfc86223548b4949398f964389b7309dfcfdcfc89125286359abb86956 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.147142575Z" level=warning msg="The image sha256:205a4d549b94d37cc0e39e08cbf8871ffe2d7e7cbb6832e26713cd69ea1e2c4f is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.147035785Z" level=warning msg="The image docker.io/kindest/kindnetd:v20230511-dc714da8 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.147341138Z" level=warning msg="The image sha256:86b6af7dd652c1b38118be1c338e9354b33469e69a218f7e290a0ca5304ad681 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.147706066Z" level=warning msg="The image sha256:c604ff157f0cff86bfa45c67c76c949deaf48d8d68560fc4c456a319af5fd8fa is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.147989139Z" level=warning msg="The image sha256:9f8f3a9f3e8a9706694dd6d7a62abd1590034454974c31cd0e21c85cf2d3a1d5 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.147847079Z" level=warning msg="The image import-2023-06-15@sha256:9d6f903c0d4bf3b145c7bbc68727251ca1abf98aed7f8d2acb9f6a10ac81e8c2 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.148038797Z" level=warning msg="The image sha256:ce18e076e9d4b4283a79ef706170486225475fc4d64253710d94780fb6ec7627 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.148805951Z" level=warning msg="The image import-2023-06-15@sha256:0202953c0b15043ca535e81d97f7062240ae66ea044b24378370d6e577782762 is not unpacked."
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.149065137Z" level=warning msg="The image docker.io/kindest/local-path-helper:v20230510-486859a6 is not unpacked."
Nov 24 15:26:47 kind-worker systemd[1]: Starting Update UTMP about System Runlevel Changes...
░░ Subject: A start job for unit systemd-update-utmp-runlevel.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-update-utmp-runlevel.service has begun execution.
░░
░░ The job identifier is 61.
Nov 24 15:26:47 kind-worker systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit systemd-update-utmp-runlevel.service has successfully entered the 'dead' state.
Nov 24 15:26:47 kind-worker systemd[1]: Finished Update UTMP about System Runlevel Changes.
░░ Subject: A start job for unit systemd-update-utmp-runlevel.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-update-utmp-runlevel.service has finished successfully.
░░
░░ The job identifier is 61.
Nov 24 15:26:47 kind-worker systemd[1]: Startup finished in 778ms.
░░ Subject: User manager start-up is now complete
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The user manager instance for user 0 has been started. All services queued
░░ for starting have been started. Note that other services might still be starting
░░ up or be started at any later time.
░░
░░ Startup of the manager took 778062 microseconds.
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.185230858Z" level=info msg="Start event monitor"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.185287012Z" level=info msg="Start snapshots syncer"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.185316974Z" level=info msg="Start cni network conf syncer for default"
Nov 24 15:26:47 kind-worker containerd[118]: time="2023-11-24T15:26:47.185341279Z" level=info msg="Start streaming server"
Nov 24 15:42:25 kind-worker systemd[1]: Starting Cleanup of Temporary Directories...
░░ Subject: A start job for unit systemd-tmpfiles-clean.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-tmpfiles-clean.service has begun execution.
░░
░░ The job identifier is 63.
Nov 24 15:42:25 kind-worker systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit systemd-tmpfiles-clean.service has successfully entered the 'dead' state.
Nov 24 15:42:25 kind-worker systemd[1]: Finished Cleanup of Temporary Directories.
░░ Subject: A start job for unit systemd-tmpfiles-clean.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-tmpfiles-clean.service has finished successfully.
░░
░░ The job identifier is 63.
Nov 24 16:05:50 kind-worker systemd[1]: Starting Load Kernel Modules...
░░ Subject: A start job for unit systemd-modules-load.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-modules-load.service has begun execution.
░░
░░ The job identifier is 67.
Nov 24 16:05:50 kind-worker systemd-modules-load[242]: Failed to insert module 'iscsi_tcp': Exec format error
Nov 24 16:05:50 kind-worker systemd-modules-load[242]: Failed to insert module 'ib_iser': Exec format error
Nov 24 16:05:50 kind-worker systemd[1]: systemd-modules-load.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit systemd-modules-load.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Nov 24 16:05:50 kind-worker systemd[1]: systemd-modules-load.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit systemd-modules-load.service has entered the 'failed' state with result 'exit-code'.
Nov 24 16:05:50 kind-worker systemd[1]: Failed to start Load Kernel Modules.
░░ Subject: A start job for unit systemd-modules-load.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit systemd-modules-load.service has finished with a failure.
░░
░░ The job identifier is 67 and the job result is failed.
root@kind-worker:/#

[BenTheElder]

Please check against https://kind.sigs.k8s.io/docs/user/known-issues/

In the meantime I recommend using docker as the node runtime for now, it's more mature. We're also adding support for nerdctl in the near future.

Will try to come back to this later, been out a lot recently.

[BenTheElder]

At a quick glance, the log dump doesn't match the supplied config given there are more nodes, was anything else different?

From containerd logs:

Nov 01 08:30:06 kind-control-plane kubelet[942]: E1101 08:30:06.341880 942 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="failed to get device for dir \"/var/lib/containerd/io.containerd.snapshotter.v1.fuse-overlayfs\": stat failed on /var/lib/containerd/io.containerd.snapshotter.v1.fuse-overlayfs with error: no such file or directory" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.fuse-overlayfs"

We're having issues with nested fuse-overlayfs on your host. Might have to force it to "native" with KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER

[aadi58002]

At a quick glance, the log dump doesn't match the supplied config given there are more nodes, was anything else different?

From containerd logs:

Nov 01 08:30:06 kind-control-plane kubelet[942]: E1101 08:30:06.341880 942 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="failed to get device for dir "/var/lib/containerd/io.containerd.snapshotter.v1.fuse-overlayfs": stat failed on /var/lib/containerd/io.containerd.snapshotter.v1.fuse-overlayfs with error: no such file or directory" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.fuse-overlayfs"

We're having issues with nested fuse-overlayfs on your host. Might have to force it to "native" with KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER

Yes the logs in second message is different from the one in the original issue. I just copy pasted one of the example from the kind main page with one control plane and some worker node for the second example. I don't have the file I used for the second comment example but it was a simple example with just control plane and some workers

[QinZhanlong]

KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER

How to set the environment variable KIND specifically when deploying a cluster ： EXPOREMENTAL CONTAINERD SNAPSHOTTER

@BenTheElder