Open alongwill opened 1 week ago
smira
commented
1 week ago @alongwill can you please check kube-proxy logs to see if it complains about nftables, I wonder Docker Desktop Linux kernel is outdated.
20.10.7 is a version released 3 years ago!
alongwill
commented
1 week ago @smira kube-proxy logs show the following nftables errors:
I1002 10:10:35.532785 1 proxier.go:1180] "Sync failed" ipFamily="IPv6" retryingTime="30s"
E1002 10:10:58.514596 1 proxier.go:1806] "nftables sync failed" err=<
/dev/stdin:78:70-88: Error: Could not process rule: No such file or directory
add rule ip kube-proxy service-2QRHZV4L-default/kubernetes/tcp/https numgen random mod 1 vmap { 0 : goto endpoint-MGIKEWY5-default/kubernetes/tcp/https__10.5.0.2/6443 }
^^^^^^^^^^^^^^^^^^^
> ipFamily="IPv4"Apologies 🤦 I pasted the docker version from my linux machine where I was testing this at the same time and it worked. The version on my Mac is as follows:
docker version
Client:
Version: 27.2.0
API version: 1.47
Go version: go1.21.13
Git commit: 3ab4256
Built: Tue Aug 27 14:14:45 2024
OS/Arch: darwin/arm64
Context: desktop-linux
Server: Docker Desktop 4.34.2 (167172)
Engine:
Version: 27.2.0
API version: 1.47 (minimum version 1.24)
Go version: go1.21.13
Git commit: 3ab5c7d
Built: Tue Aug 27 14:15:41 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.20
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0N.B. Creating a cluster on my mac with the same docker version and talosctl 1.7.6 works.
| Summary: | Arch | talosctl | Working? |
|---|---|---|---|
🍎 arm64 |
1.7.6 | ✅ | |
🍎 arm64 |
1.8.0 | ❌ | |
🐧 x86_64 |
1.8.0 | ✅ |
smira
commented
1 week ago yeah, I guess not much we can do if we want to stay consistent, but Kubernetes 1.31 defaults to nftables backend in Talos
so you can bring up 1.8.0 with Kubernetes 1.30 for example, and it would probably work, but this is something beyond our control (Docker Desktop kernel).
Bug Report
Description
When creating a cluster with
talosctland docker,corednsdoes not startView logs
Logs
corednsPods are not ready.View pod statuses
corednsPod logsView coredns pod logs
View coredns pod events
kubectl describe -n kube-system po coredns-68d75fd545-bwtqm ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 16m default-scheduler no nodes available to schedule pods Warning FailedScheduling 16m default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. Normal Scheduled 15m default-scheduler Successfully assigned kube-system/coredns-68d75fd545-bwtqm to talos-default-worker-1 Warning FailedCreatePodSandBox 15m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "73940f0a8b5724ae1fe13c750a886d2a0e9c39ac254ee82c6cc08462dee1cb3f": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory Normal Pulling 15m kubelet Pulling image "registry.k8s.io/coredns/coredns:v1.11.3" Normal Pulled 15m kubelet Successfully pulled image "registry.k8s.io/coredns/coredns:v1.11.3" in 5.508s (5.508s including waiting). Image size: 16948420 bytes. Normal Created 15m kubelet Created container coredns Normal Started 15m kubelet Started container coredns Warning Unhealthy 41s (x103 over 15m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503Environment
talosctl version --nodes <problematic nodes>]kubectl version --short]Server: Docker Engine - Community Engine: Version: 20.10.7 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: b0f5bc3 Built: Wed Jun 2 11:54:48 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.6 GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d runc: Version: 1.0.0-rc95 GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 docker-init: Version: 0.19.0 GitCommit: de40ad0