kim for osx kind-backed clusters?

djcp commented 2 years ago

So, as an experiment I've integrated kim with tilt using a kind backend. This works perfectly on ubuntu, and has allowed us to shave a pretty hefty chunk off build times (25 to 40%) by avoiding the image registry one would normally need for a tilt / kind / docker project.

However, on osx using kind and docker-desktop-for-mac, kim fails with the error:

 level=fatal msg="failed to get status: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.24.0.2:1234: i/o timeout\""

I know Smarter automatic-ish bootstrap for non-k3s installations (think EKS support) is on the roadmap, and I assume this would fall into that bucket.

It appears to install the infrastructure needed by kim correctly, but when it comes to actually interacting with it - builds, kim image ls, etc, it fails with the error above. Any suggestions about how to get beyond this error are very welcome!

dweomer commented 2 years ago

@djcp I left this functionality hidden because I only expected Rancher Desktop to need it: https://github.com/rancher/kim/blob/v0.1.0-beta.7/pkg/client/builder/install.go#L30

You will need to make sure that you have a port-forward setup from the host to the builder node container in the guest (automatic on localhost I think with docker-desktop) and then you can run kim builder install --endpoint-addr <host>. Include --force to reinstall (you won't lose any images) over the top of an existing installation.

dweomer commented 2 years ago

cat ~/.kind/kim.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 1233
    hostPort: 1233
    protocol: TCP
  - containerPort: 1234
    hostPort: 1234
    protocol: TCP

kind create cluster --config ~/.kind/kim.yaml:

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊

kubectl get node -o wide:

NAME                 STATUS     ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION      CONTAINER-RUNTIME
kind-control-plane   NotReady   control-plane,master   18s   v1.21.1   172.19.0.2    <none>        Ubuntu 21.04   4.19.121-linuxkit   containerd://1.5.2

kim builder install --endpoint-addr 127.0.0.1:

INFO[0000] Applying node-role `builder` to `kind-control-plane` 
INFO[0000] Asserting namespace `kube-image`             
INFO[0000] Asserting TLS secrets                        
INFO[0000] Asserting service/endpoints                  
INFO[0000] Installing builder daemon                    
INFO[0000] Waiting on builder daemon availability...    
INFO[0006] Waiting on builder daemon availability...    
INFO[0011] Waiting on builder daemon availability...    
INFO[0017] Waiting on builder daemon availability...

kubectl describe -n kube-image service/builder (notice the images.cattle.io/endpoint-override annotation):

Name:                     builder
Namespace:                kube-image
Labels:                   app.kubernetes.io/managed-by=kim
Annotations:              images.cattle.io/endpoint-override: 127.0.0.1
Selector:                 app.kubernetes.io/component=builder,app.kubernetes.io/name=kim
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.70.153
IPs:                      10.96.70.153
Port:                     buildkit  1234/TCP
TargetPort:               1234/TCP
NodePort:                 buildkit  31299/TCP
Endpoints:                172.19.0.2:1234
Port:                     kim  1233/TCP
TargetPort:               1233/TCP
NodePort:                 kim  31366/TCP
Endpoints:                172.19.0.2:1233
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

kim image ls -a:

IMAGE                                TAG                  IMAGE ID            SIZE
k8s.gcr.io/etcd                      3.4.13-0             0369cf4303ffd       86.7MB
k8s.gcr.io/kube-proxy                v1.21.1              0e124fb3c695b       133MB
k8s.gcr.io/kube-scheduler            v1.21.1              1248d2d503d37       51.9MB
k8s.gcr.io/coredns/coredns           v1.8.0               296a6d5035e2d       12.9MB
kindest/kindnetd                     v20210326-1e038dc5   6de166512aa22       54MB
k8s.gcr.io/kube-apiserver            v1.21.1              94ffe308aeff9       127MB
k8s.gcr.io/kube-controller-manager   v1.21.1              96a295389d472       121MB
k8s.gcr.io/build-image/debian-base   v2.1.0               c7c6c86897b63       21.1MB
rancher/local-path-provisioner       v0.0.14              e422121c9c5f9       13.4MB
k8s.gcr.io/pause                     3.5                  ed210e3e4a5ba       301kB
moby/buildkit                        v0.8.3               cf14c5e88c0eb       56.5MB
rancher/kim                          v0.1.0-beta.4        1f246fbed18ae       13.7MB

rancher / kim

kim for osx kind-backed clusters? #85