search

abiosoft / colima

Container runtimes on macOS (and Linux) with minimal setup
MIT License
17.75k stars 365 forks source link

docker+k3s: no Node available, probably due to `--docker is no longer supported` from k3s #957

Closed choznerol closed 5 months ago

choznerol commented 5 months ago

Description

Update: Use k3s >= v1.24.3+k3s1 should fix the --docker is no longer supported from k3s:

This release restores use of the --docker flag to the v1.24 branch. See docs/adrs/cri-dockerd.md for more information.

See https://github.com/abiosoft/colima/issues/957#issuecomment-1890410527

There isn't any node available in the k3s cluster with the default --runtime docker.

I noticed a k3s log line complaining that the --docker flag is no longer supported:

Jan 13 09:37:28 colima-dkik k3s[2842]: time="2024-01-13T09:37:28Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: --docker is no longer supported; to continue using docker, install cri-dockerd and set --container-runtime-endpoint"

Full output of sudo systemctl status k3s --no-pager -l ``` ychou@colima-dkik:~$ sudo systemctl status k3s --no-pager -l ● k3s.service - Lightweight Kubernetes Loaded: loaded (/etc/systemd/system/k3s.service; disabled; preset: enabled) Active: active (running) since Sat 2024-01-13 09:23:02 UTC; 14min ago Docs: https://k3s.io Process: 2838 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS) Process: 2840 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 2841 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 2842 (k3s-server) Tasks: 23 Memory: 513.8M CPU: 15min 29.476s CGroup: /system.slice/k3s.service └─2842 "/usr/local/bin/k3s server" Jan 13 09:37:22 colima-dkik k3s[2842]: time="2024-01-13T09:37:22Z" level=info msg="Waiting for control-plane node agent startup" Jan 13 09:37:23 colima-dkik k3s[2842]: time="2024-01-13T09:37:23Z" level=info msg="Waiting for control-plane node agent startup" Jan 13 09:37:23 colima-dkik k3s[2842]: time="2024-01-13T09:37:23Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: --docker is no longer supported; to continue using docker, install cri-dockerd and set --container-runtime-endpoint" Jan 13 09:37:24 colima-dkik k3s[2842]: time="2024-01-13T09:37:24Z" level=info msg="Waiting for control-plane node agent startup" Jan 13 09:37:25 colima-dkik k3s[2842]: time="2024-01-13T09:37:25Z" level=info msg="Waiting for control-plane node agent startup" Jan 13 09:37:26 colima-dkik k3s[2842]: time="2024-01-13T09:37:26Z" level=info msg="Waiting for control-plane node agent startup" Jan 13 09:37:27 colima-dkik k3s[2842]: time="2024-01-13T09:37:27Z" level=info msg="Waiting for control-plane node agent startup" Jan 13 09:37:28 colima-dkik k3s[2842]: time="2024-01-13T09:37:28Z" level=info msg="Waiting for control-plane node agent startup" Jan 13 09:37:28 colima-dkik k3s[2842]: time="2024-01-13T09:37:28Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: --docker is no longer supported; to continue using docker, install cri-dockerd and set --container-runtime-endpoint" Jan 13 09:37:29 colima-dkik k3s[2842]: time="2024-01-13T09:37:29Z" level=info msg="Waiting for control-plane node agent startup" ```

... which is probably passed here:

https://github.com/abiosoft/colima/blob/4e52450080340a2861edf647f460563a21e011d4/environment/container/kubernetes/k3s.go#L154-L155

Version

ychou@colima-cdik:~$ colima version
colima version 0.6.7
git commit: ba1be00e9aec47f2c1ffdacfb7e428e465f0b58a
ychou@colima-cdik:~$ limactl --version
limactl version 0.19.0
ychou@colima-cdik:~$ qemu-img --version
qemu-img version 8.1.3
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

also the k3s version:

ychou@colima-cdik:~$ k3s --version
k3s version v1.24.3+k3s1 (990ba0e8)
go version go1.18.1

Operating System

Output of colima status

$ colima -p dkik status
INFO[0000] colima [profile=dkik] is running using QEMU
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/ychou/.colima/dkik/docker.sock
INFO[0000] kubernetes: enable

Reproduction Steps

  1. colima -p dkik start --runtime docker --kubernetes # Using --runtime containerd here can work around the issue

  2. Try to list k8s node or run a pod

colima -p dkik ssh

# No node avaialble!
ychou@colima-dkik:~$ kubectl get nodes
No resources found

# Pod forever 'Pending'
ychou@colima-dkik:~$ kubectl run pod-wont-schedule --image nginx
pod/pod-wont-schedule created
ychou@colima-dkik:~$ kubectl get po
NAME                READY   STATUS    RESTARTS   AGE
pod-wont-schedule   0/1     Pending   0          6s
ychou@colima-dkik:~$ kubectl get po
NAME                READY   STATUS    RESTARTS   AGE
pod-wont-schedule   0/1     Pending   0          6m7s

# Same for the system pods
ychou@colima-dkik:~$ kubectl get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
metrics-server-668d979685-9zh46           0/1     Pending   0          7m54s
coredns-b96499967-2lhkv                   0/1     Pending   0          7m54s
local-path-provisioner-7b7dc8d6f5-7djcw   0/1     Pending   0          7m54s

Expected behaviour

Have some node available and can schedule pod. This is true with --runtime containerd:

$ colima -p cdik start --runtime containerd --kubernetes
...

$ colima -p cdik status
INFO[0000] colima [profile=cdik] is running using QEMU
INFO[0000] arch: aarch64
INFO[0000] runtime: containerd
INFO[0000] mountType: sshfs
INFO[0000] kubernetes: enabled

$ colima -p cdik ssh

ychou@colima-cdik:~$ kubectl get nodes
NAME          STATUS   ROLES                  AGE   VERSION
colima-cdik   Ready    control-plane,master   38m   v1.24.3+k3s1

ychou@colima-cdik:~$ kubectl run mynginx --image nginx
pod/mynginx created

ychou@colima-cdik:~$ kubectl get po
NAME        READY   STATUS              RESTARTS   AGE
mynginx     1/1     Running             0          119s

Additional context

No error events from the k3s cluster:

k get events -A

ychou@colima-dkik:~$ kubectl get events -A
NAMESPACE     LAST SEEN   TYPE     REASON              OBJECT                                         MESSAGE
kube-system   17m         Normal   ApplyingManifest    addon/ccm                                      Applying manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml"
kube-system   17m         Normal   AppliedManifest     addon/ccm                                      Applied manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/coredns                                  Applying manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml"
kube-system   17m         Normal   AppliedManifest     addon/coredns                                  Applied manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/local-storage                            Applying manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml"
kube-system   17m         Normal   AppliedManifest     addon/local-storage                            Applied manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/aggregated-metrics-reader                Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
kube-system   17m         Normal   AppliedManifest     addon/aggregated-metrics-reader                Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/auth-delegator                           Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
kube-system   17m         Normal   AppliedManifest     addon/auth-delegator                           Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/auth-reader                              Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
kube-system   17m         Normal   AppliedManifest     addon/auth-reader                              Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/metrics-apiservice                       Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
kube-system   17m         Normal   AppliedManifest     addon/metrics-apiservice                       Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/metrics-server-deployment                Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
kube-system   17m         Normal   AppliedManifest     addon/metrics-server-deployment                Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/metrics-server-service                   Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
kube-system   17m         Normal   AppliedManifest     addon/metrics-server-service                   Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/resource-reader                          Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
kube-system   17m         Normal   AppliedManifest     addon/resource-reader                          Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
kube-system   17m         Normal   ApplyingManifest    addon/rolebindings                             Applying manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
kube-system   17m         Normal   AppliedManifest     addon/rolebindings                             Applied manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
kube-system   16m         Normal   ScalingReplicaSet   deployment/coredns                             Scaled up replica set coredns-b96499967 to 1
kube-system   16m         Normal   ScalingReplicaSet   deployment/local-path-provisioner              Scaled up replica set local-path-provisioner-7b7dc8d6f5 to 1
kube-system   16m         Normal   ScalingReplicaSet   deployment/metrics-server                      Scaled up replica set metrics-server-668d979685 to 1
kube-system   16m         Normal   SuccessfulCreate    replicaset/metrics-server-668d979685           Created pod: metrics-server-668d979685-9zh46
kube-system   16m         Normal   SuccessfulCreate    replicaset/local-path-provisioner-7b7dc8d6f5   Created pod: local-path-provisioner-7b7dc8d6f5-7djcw
kube-system   16m         Normal   SuccessfulCreate    replicaset/coredns-b96499967                   Created pod: coredns-b96499967-2lhkv

No error from k3s log

sudo journalctl /usr/local/bin/k3s

ychou@colima-dkik:~$ sudo journalctl --no-pager -l /usr/local/bin/k3s
Jan 13 09:22:57 colima-dkik k3s[2842]: time="2024-01-13T09:22:57Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Jan 13 09:22:57 colima-dkik k3s[2842]: time="2024-01-13T09:22:57Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/230cb7b95d48672d3d564dfde9c7b6230bccc82b2a3298f565dc3cca2a06b2eb"
Jan 13 09:22:58 colima-dkik k3s[2842]: time="2024-01-13T09:22:58Z" level=info msg="Found ip 192.168.5.8 from iface eth0"
Jan 13 09:22:58 colima-dkik k3s[2842]: time="2024-01-13T09:22:58Z" level=info msg="Starting k3s v1.24.3+k3s1 (990ba0e8)"

choznerol commented 5 months ago

Just confirmed that --docker is no longer supported can be fixed by configuring k3s >= v1.24.3+k3s1:

This release restores use of the --docker flag to the v1.24 branch. See docs/adrs/cri-dockerd.md for more information.

detail ``` $ colima -p dkik-1-24-4 start --runtime docker --kubernetes --edit # Update k3s version to >= v1.24.4+k3s1 INFO[0000] editing in lvim from $EDITOR environment variable INFO[0013] starting colima [profile=dkik-1-24-4] INFO[0013] runtime: docker+k3s INFO[0013] creating and starting ... context=vm INFO[0105] provisioning ... context=docker INFO[0105] starting ... context=docker INFO[0106] provisioning ... context=kubernetes INFO[0107] downloading and installing ... context=kubernetes INFO[0608] loading oci images ... context=kubernetes INFO[0618] starting ... context=kubernetes INFO[0624] updating config ... context=kubernetes INFO[0625] Switched to context "colima-dkik-1-24-4". context=kubernetes INFO[0626] done $ colima -p dkik-1-24-4 ssh ychou@colima-dkik-1-24-4:/Users/ychou$ kubectl get nodes NAME STATUS ROLES AGE VERSION colima-dkik-1-24-4 Ready control-plane,master 31s v1.24.4+k3s1 ```

I'll close this since there is probably no actionable item (e.g. add warning) form Colima side 🙏

P.s. I'll personally default to containerd from now on 👍