search

rancher / rke2

https://docs.rke2.io/
Apache License 2.0
1.54k stars 267 forks source link

RKE2 not starting on fresh ubuntu 22.04 #4057

Closed mowoe closed 1 year ago

mowoe commented 1 year ago

Environmental Info: RKE2 Version: rke2 version v1.24.11+rke2r1 (23bdc987de9658eb80a5e03862ab34efdecf6d4e)

Node(s) CPU architecture, OS, and Version: Linux k8s-rdntz-eu-lej-1-1 5.15.0-60-generic #66-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 1 server, 1 agent, not even getting as far as one server starting

Describe the bug:

Steps To Reproduce:

  cluster-cidr: "10.42.0.0/16,2a0e:8f02:f000:dede:fefe::/70"
  service-cidr: "10.43.0.0/16,2a0e:8f02:f000:dede:fdfd::/70"

Expected behavior: RKE2 being able to start up normally

Additional context / logs:

containerd log ``` time="2023-03-25T22:38:58.188533812Z" level=info msg="starting containerd" revision=e2856823e51b3f3c5435e87ddd79ba4fbd8f3daa version=v1.6.15-k3s1 time="2023-03-25T22:38:58.209769936Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1 time="2023-03-25T22:38:58.209889206Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.211909434Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.15.0-60-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.211926284Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.212052224Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.212061864Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.212069684Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured" time="2023-03-25T22:38:58.212075594Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.212128894Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.212302854Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.fuse-overlayfs\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.212359184Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.stargz\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.247592330Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.247687080Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1 time="2023-03-25T22:38:58.247698150Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1 time="2023-03-25T22:38:58.247745100Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured" time="2023-03-25T22:38:58.247752470Z" level=info msg="metadata content store policy set" policy=shared time="2023-03-25T22:38:58.327288852Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1 time="2023-03-25T22:38:58.327308501Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1 time="2023-03-25T22:38:58.327318591Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1 time="2023-03-25T22:38:58.327339261Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327350821Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327362561Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327376841Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327388251Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327398691Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327408111Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327417351Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327427101Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1 time="2023-03-25T22:38:58.327506161Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2 time="2023-03-25T22:38:58.327570831Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1 time="2023-03-25T22:38:58.327741111Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1 time="2023-03-25T22:38:58.327759141Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327768851Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1 time="2023-03-25T22:38:58.327802551Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327812561Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327821271Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327831551Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327839821Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327847891Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327855861Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327864411Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327872791Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1 time="2023-03-25T22:38:58.327952381Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327962021Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327969911Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.327978181Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1 time="2023-03-25T22:38:58.327988461Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1 time="2023-03-25T22:38:58.327994591Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1 time="2023-03-25T22:38:58.328009161Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin" time="2023-03-25T22:38:58.328042401Z" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1 time="2023-03-25T22:38:58.328197231Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:10010 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:index.docker.io/rancher/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:true EnableUnprivilegedICMP:true} ContainerdRootDir:/var/lib/rancher/rke2/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/var/lib/rancher/rke2/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}" time="2023-03-25T22:38:58.328232421Z" level=info msg="Connect containerd service" time="2023-03-25T22:38:58.328256301Z" level=info msg="Get image filesystem path \"/var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs\"" time="2023-03-25T22:38:58.328816389Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config" time="2023-03-25T22:38:58.330326438Z" level=info msg=serving... address=/run/k3s/containerd/containerd.sock.ttrpc time="2023-03-25T22:38:58.330357978Z" level=info msg=serving... address=/run/k3s/containerd/containerd.sock time="2023-03-25T22:38:58.330373688Z" level=info msg="containerd successfully booted in 0.142492s" time="2023-03-25T22:38:58.330955817Z" level=info msg="Start subscribing containerd event" time="2023-03-25T22:38:58.331035747Z" level=info msg="Start recovering state" time="2023-03-25T22:38:58.331143227Z" level=info msg="Start event monitor" time="2023-03-25T22:38:58.331154987Z" level=info msg="Start snapshots syncer" time="2023-03-25T22:38:58.331166187Z" level=info msg="Start cni network conf syncer for default" time="2023-03-25T22:38:58.331199917Z" level=info msg="Start streaming server" time="2023-03-25T22:38:59.250887564Z" level=info msg="PullImage \"index.docker.io/rancher/rke2-cloud-provider:v1.26.1-build20230210\"" time="2023-03-25T22:39:16.999819333Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/rke2-cloud-provider:v1.26.1-build20230210,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:17.277475504Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:3e9d08b8ec142a0ec9da5f8591dab5ea2ba54deda04385cee8cfbe2acbf49441,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:17.545383442Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/rancher/rke2-cloud-provider:v1.26.1-build20230210,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:17.883240315Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/rke2-cloud-provider@sha256:9d54d0e0c6ffa36854255d18347fb85b1a34aee0a82d17182ec11b0d704892de,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:17.883662764Z" level=info msg="PullImage \"index.docker.io/rancher/rke2-cloud-provider:v1.26.1-build20230210\" returns image reference \"sha256:3e9d08b8ec142a0ec9da5f8591dab5ea2ba54deda04385cee8cfbe2acbf49441\"" time="2023-03-25T22:39:17.885290504Z" level=info msg="PullImage \"index.docker.io/rancher/hardened-etcd:v3.5.4-k3s1-build20221011\"" time="2023-03-25T22:39:46.856045873Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/hardened-etcd:v3.5.4-k3s1-build20221011,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:47.165336749Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:df303eb7d56bcb2e1f0f74bbc26fb0fd0893c21147790b748c41114e7e224219,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:47.589782105Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/rancher/hardened-etcd:v3.5.4-k3s1-build20221011,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:48.082238242Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/hardened-etcd@sha256:abb0674e4f1462153f3542ea90c0272988e024d304fe9e1f3267643d744caaab,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:39:48.082773892Z" level=info msg="PullImage \"index.docker.io/rancher/hardened-etcd:v3.5.4-k3s1-build20221011\" returns image reference \"sha256:df303eb7d56bcb2e1f0f74bbc26fb0fd0893c21147790b748c41114e7e224219\"" time="2023-03-25T22:39:48.083924982Z" level=info msg="PullImage \"index.docker.io/rancher/hardened-kubernetes:v1.24.11-rke2r1-build20230228\"" time="2023-03-25T22:40:28.389715769Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/hardened-kubernetes:v1.24.11-rke2r1-build20230228,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:40:28.724379960Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:ea7f73e06814a57be25ce3e8947b47632c06e909ee07f475dac124a1c48b176b,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:40:29.103988073Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/rancher/hardened-kubernetes:v1.24.11-rke2r1-build20230228,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:40:29.578825159Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/hardened-kubernetes@sha256:1ac2854d76433bb894cc639fb28ce67c53da3a268384d6b353130dc9102425e9,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:40:29.579563121Z" level=info msg="PullImage \"index.docker.io/rancher/hardened-kubernetes:v1.24.11-rke2r1-build20230228\" returns image reference \"sha256:ea7f73e06814a57be25ce3e8947b47632c06e909ee07f475dac124a1c48b176b\"" time="2023-03-25T22:40:35.156427325Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-k8s-rdntz-eu-lej-1-1,Uid:e18aa5e5b83a5a3c56d78e4054612394,Namespace:kube-system,Attempt:0,}" time="2023-03-25T22:40:56.539001414Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:40:57.309279591Z" level=info msg="ImageCreate event &ImageCreate{Name:sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:40:57.543461738Z" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/rancher/pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:40:57.901455281Z" level=info msg="ImageCreate event &ImageCreate{Name:docker.io/rancher/pause@sha256:036d575e82945c112ef84e4585caff3648322a2f9ed4c3a6ce409dd10abc4f34,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}" time="2023-03-25T22:41:01.077081049Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2023-03-25T22:41:01.077141759Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2023-03-25T22:41:01.077150579Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2023-03-25T22:41:01.077268528Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/1439984b0d9038a49d5cd5b534c53e2a5a7262d0ab984820d057a3149bcbbb73 pid=1415 runtime=io.containerd.runc.v2 time="2023-03-25T22:41:01.170409153Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-k8s-rdntz-eu-lej-1-1,Uid:e18aa5e5b83a5a3c56d78e4054612394,Namespace:kube-system,Attempt:0,} returns sandbox id \"1439984b0d9038a49d5cd5b534c53e2a5a7262d0ab984820d057a3149bcbbb73\"" time="2023-03-25T22:41:01.174533739Z" level=info msg="CreateContainer within sandbox \"1439984b0d9038a49d5cd5b534c53e2a5a7262d0ab984820d057a3149bcbbb73\" for container &ContainerMetadata{Name:etcd,Attempt:0,}" time="2023-03-25T22:41:05.602249512Z" level=info msg="CreateContainer within sandbox \"1439984b0d9038a49d5cd5b534c53e2a5a7262d0ab984820d057a3149bcbbb73\" for &ContainerMetadata{Name:etcd,Attempt:0,} returns container id \"24b41b80f60e7426dd2de4880de96e348b691b5cd13a63997d29f10592b12791\"" time="2023-03-25T22:41:05.602759149Z" level=info msg="StartContainer for \"24b41b80f60e7426dd2de4880de96e348b691b5cd13a63997d29f10592b12791\"" time="2023-03-25T22:41:06.214780117Z" level=info msg="StartContainer for \"24b41b80f60e7426dd2de4880de96e348b691b5cd13a63997d29f10592b12791\" returns successfully" time="2023-03-25T22:42:12.466625710Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-k8s-rdntz-eu-lej-1-1,Uid:4874a08227e8932676b83ca998a390f3,Namespace:kube-system,Attempt:0,}" time="2023-03-25T22:42:12.681683833Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1 time="2023-03-25T22:42:12.681760523Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1 time="2023-03-25T22:42:12.681769433Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1 time="2023-03-25T22:42:12.681917122Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a pid=1503 runtime=io.containerd.runc.v2 time="2023-03-25T22:42:12.767833464Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-k8s-rdntz-eu-lej-1-1,Uid:4874a08227e8932676b83ca998a390f3,Namespace:kube-system,Attempt:0,} returns sandbox id \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\"" time="2023-03-25T22:42:12.771693262Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for container &ContainerMetadata{Name:kube-apiserver,Attempt:0,}" time="2023-03-25T22:42:13.031887700Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for &ContainerMetadata{Name:kube-apiserver,Attempt:0,} returns container id \"44b057f6ef19bdbe5810b947bea6850912f9746c1470a8f110d25fdf26ef29e7\"" time="2023-03-25T22:42:13.032427978Z" level=info msg="StartContainer for \"44b057f6ef19bdbe5810b947bea6850912f9746c1470a8f110d25fdf26ef29e7\"" time="2023-03-25T22:42:13.178503837Z" level=info msg="StartContainer for \"44b057f6ef19bdbe5810b947bea6850912f9746c1470a8f110d25fdf26ef29e7\" returns successfully" time="2023-03-25T22:42:13.415806141Z" level=info msg="shim disconnected" id=44b057f6ef19bdbe5810b947bea6850912f9746c1470a8f110d25fdf26ef29e7 time="2023-03-25T22:42:13.415895540Z" level=warning msg="cleaning up after shim disconnected" id=44b057f6ef19bdbe5810b947bea6850912f9746c1470a8f110d25fdf26ef29e7 namespace=k8s.io time="2023-03-25T22:42:13.415906830Z" level=info msg="cleaning up dead shim" time="2023-03-25T22:42:13.441285299Z" level=warning msg="cleanup warnings time=\"2023-03-25T22:42:13Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=1589 runtime=io.containerd.runc.v2\n" time="2023-03-25T22:42:14.303543518Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for container &ContainerMetadata{Name:kube-apiserver,Attempt:1,}" time="2023-03-25T22:42:15.816011278Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for &ContainerMetadata{Name:kube-apiserver,Attempt:1,} returns container id \"7dcc53be42435037fdc447456b3fd51c9da90065cc9fb8454a24e9804a70bc26\"" time="2023-03-25T22:42:15.816359077Z" level=info msg="StartContainer for \"7dcc53be42435037fdc447456b3fd51c9da90065cc9fb8454a24e9804a70bc26\"" time="2023-03-25T22:42:16.106250318Z" level=info msg="StartContainer for \"7dcc53be42435037fdc447456b3fd51c9da90065cc9fb8454a24e9804a70bc26\" returns successfully" time="2023-03-25T22:42:16.465209319Z" level=info msg="shim disconnected" id=7dcc53be42435037fdc447456b3fd51c9da90065cc9fb8454a24e9804a70bc26 time="2023-03-25T22:42:16.465269138Z" level=warning msg="cleaning up after shim disconnected" id=7dcc53be42435037fdc447456b3fd51c9da90065cc9fb8454a24e9804a70bc26 namespace=k8s.io time="2023-03-25T22:42:16.465281588Z" level=info msg="cleaning up dead shim" time="2023-03-25T22:42:16.479071325Z" level=warning msg="cleanup warnings time=\"2023-03-25T22:42:16Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=1655 runtime=io.containerd.runc.v2\n" time="2023-03-25T22:42:17.309264380Z" level=info msg="RemoveContainer for \"44b057f6ef19bdbe5810b947bea6850912f9746c1470a8f110d25fdf26ef29e7\"" time="2023-03-25T22:42:18.117252912Z" level=info msg="RemoveContainer for \"44b057f6ef19bdbe5810b947bea6850912f9746c1470a8f110d25fdf26ef29e7\" returns successfully" time="2023-03-25T22:42:37.717524201Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for container &ContainerMetadata{Name:kube-apiserver,Attempt:2,}" time="2023-03-25T22:42:37.923682159Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for &ContainerMetadata{Name:kube-apiserver,Attempt:2,} returns container id \"fc033f2bd8bf46668d15a7fb6f77b12730d36c79aaa1ee58af0a9a0026dd1078\"" time="2023-03-25T22:42:37.924149548Z" level=info msg="StartContainer for \"fc033f2bd8bf46668d15a7fb6f77b12730d36c79aaa1ee58af0a9a0026dd1078\"" time="2023-03-25T22:42:38.089951929Z" level=info msg="StartContainer for \"fc033f2bd8bf46668d15a7fb6f77b12730d36c79aaa1ee58af0a9a0026dd1078\" returns successfully" time="2023-03-25T22:42:38.163153384Z" level=info msg="shim disconnected" id=fc033f2bd8bf46668d15a7fb6f77b12730d36c79aaa1ee58af0a9a0026dd1078 time="2023-03-25T22:42:38.163290714Z" level=warning msg="cleaning up after shim disconnected" id=fc033f2bd8bf46668d15a7fb6f77b12730d36c79aaa1ee58af0a9a0026dd1078 namespace=k8s.io time="2023-03-25T22:42:38.163305054Z" level=info msg="cleaning up dead shim" time="2023-03-25T22:42:38.180862420Z" level=warning msg="cleanup warnings time=\"2023-03-25T22:42:38Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=1720 runtime=io.containerd.runc.v2\n" time="2023-03-25T22:42:38.339793473Z" level=info msg="RemoveContainer for \"7dcc53be42435037fdc447456b3fd51c9da90065cc9fb8454a24e9804a70bc26\"" time="2023-03-25T22:42:38.431190743Z" level=info msg="RemoveContainer for \"7dcc53be42435037fdc447456b3fd51c9da90065cc9fb8454a24e9804a70bc26\" returns successfully" time="2023-03-25T22:43:06.716269264Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for container &ContainerMetadata{Name:kube-apiserver,Attempt:3,}" time="2023-03-25T22:43:07.009637428Z" level=info msg="CreateContainer within sandbox \"5568be6578deb4695f80c7786c5d0abebf73948f33bcbb387f982b62c3e4db0a\" for &ContainerMetadata{Name:kube-apiserver,Attempt:3,} returns container id \"b96d9b5ab4e7de7d397c8afd75150d8950fa421714df9c09e59803dbc61555fc\"" time="2023-03-25T22:43:07.010058937Z" level=info msg="StartContainer for \"b96d9b5ab4e7de7d397c8afd75150d8950fa421714df9c09e59803dbc61555fc\"" time="2023-03-25T22:43:07.148466645Z" level=info msg="StartContainer for \"b96d9b5ab4e7de7d397c8afd75150d8950fa421714df9c09e59803dbc61555fc\" returns successfully" time="2023-03-25T22:43:07.309195805Z" level=info msg="shim disconnected" id=b96d9b5ab4e7de7d397c8afd75150d8950fa421714df9c09e59803dbc61555fc time="2023-03-25T22:43:07.309409984Z" level=warning msg="cleaning up after shim disconnected" id=b96d9b5ab4e7de7d397c8afd75150d8950fa421714df9c09e59803dbc61555fc namespace=k8s.io time="2023-03-25T22:43:07.309427504Z" level=info msg="cleaning up dead shim" time="2023-03-25T22:43:07.333182703Z" level=warning msg="cleanup warnings time=\"2023-03-25T22:43:07Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=1788 runtime=io.containerd.runc.v2\n" time="2023-03-25T22:43:07.379894583Z" level=info msg="RemoveContainer for \"fc033f2bd8bf46668d15a7fb6f77b12730d36c79aaa1ee58af0a9a0026dd1078\"" time="2023-03-25T22:43:07.430233023Z" level=info msg="RemoveContainer for \"fc033f2bd8bf46668d15a7fb6f77b12730d36c79aaa1ee58af0a9a0026dd1078\" returns successfully" ```

RKE2 Log: pastebin

mowoe commented 1 year ago

The problem was the service cidr being too large... Is this documented anywhere? (I realise that this is a kubeadm problem and not a RKE2 thing).

root@k8s-rdntz-eu-lej-1-1:~# /var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock logs 66062edb7c9d2
I0326 09:52:51.505510       1 server.go:558] external host was not specified, using 10.0.0.2
E0326 09:52:51.505741       1 run.go:74] "command failed" err="specified --service-cluster-ip-range[1] is too large; for 128-bit addresses, the mask must be >= 108"
root@k8s-rdntz-eu-lej-1-1:~#

Anyway, closing this.