Closed nirs closed 1 month ago
nirs
commented
1 month ago I think we can add a job using this, creating 2 tiny clusters and accessing them using the ip address on the vmnet network. Such test will ensure that upstream lima change will not break our use case.
nirs
commented
1 month ago This can also a new integration test.
% cat create-clusters.sh
#!/bin/sh
limactl start --name c1 template://k8s-vmnet &
limactl start --name c2 template://k8s-vmnet &
wait
% ./create-clusters.sh
INFO[0000] Starting the instance "c2" with VM driver "vz"
INFO[0000] Starting the instance "c1" with VM driver "vz"
INFO[0000] Attempting to download the image arch=aarch64 digest= location="https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img"
INFO[0000] Using cache "/Users/nsoffer/Library/Caches/lima/download/by-url-sha256/002fbe468673695a2206b26723b1a077a71629001a5b94efd8ea1580e1c3dd06/data"
INFO[0000] Converting "/Users/nsoffer/.lima/c2/basedisk" (qcow2) to a raw disk "/Users/nsoffer/.lima/c2/diffdisk"
INFO[0000] Attempting to download the image arch=aarch64 digest= location="https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img"
INFO[0000] Using cache "/Users/nsoffer/Library/Caches/lima/download/by-url-sha256/002fbe468673695a2206b26723b1a077a71629001a5b94efd8ea1580e1c3dd06/data"
INFO[0000] Converting "/Users/nsoffer/.lima/c1/basedisk" (qcow2) to a raw disk "/Users/nsoffer/.lima/c1/diffdisk"
3.50 GiB / 3.50 GiB [-------------------------------------] 100.00% 193.31 MiB/s
INFO[0018] Expanding to 100GiB
3.50 GiB / 3.50 GiB [-------------------------------------] 100.00% 193.27 MiB/s
INFO[0018] Expanding to 100GiB
INFO[0019] Attempting to download the nerdctl archive arch=aarch64 digest="sha256:77c747f09853ee3d229d77e8de0dd3c85622537d82be57433dc1fca4493bab95" location="https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-full-1.7.6-linux-arm64.tar.gz"
INFO[0019] Using cache "/Users/nsoffer/Library/Caches/lima/download/by-url-sha256/21cc8dfa548ea8a678135bd6984c9feb9f8a01901d10b11bb491f6f4e7537158/data"
INFO[0019] Attempting to download the nerdctl archive arch=aarch64 digest="sha256:77c747f09853ee3d229d77e8de0dd3c85622537d82be57433dc1fca4493bab95" location="https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-full-1.7.6-linux-arm64.tar.gz"
INFO[0019] Using cache "/Users/nsoffer/Library/Caches/lima/download/by-url-sha256/21cc8dfa548ea8a678135bd6984c9feb9f8a01901d10b11bb491f6f4e7537158/data"
INFO[0020] [hostagent] TCP port forwarding is disabled (except for SSH)
INFO[0020] [hostagent] TCP port forwarding is disabled (except for SSH)
WARN[0020] [hostagent] GRPC port forwarding is experimental
WARN[0020] [hostagent] GRPC port forwarding is experimental
INFO[0020] [hostagent] hostagent socket created at /Users/nsoffer/.lima/c1/ha.sock
INFO[0020] [hostagent] hostagent socket created at /Users/nsoffer/.lima/c2/ha.sock
INFO[0020] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/nsoffer/.lima/c1/serial*.log")
INFO[0020] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/nsoffer/.lima/c2/serial*.log")
INFO[0020] [hostagent] new connection from to
INFO[0020] [hostagent] new connection from to
INFO[0020] SSH Local Port: 62794
INFO[0020] SSH Local Port: 62796
INFO[0020] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0020] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0020] [hostagent] [VZ] - vm state change: running
INFO[0020] [hostagent] [VZ] - vm state change: running
INFO[0030] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0030] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0044] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0044] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0045] [hostagent] The essential requirement 1 of 2 is satisfied
INFO[0045] [hostagent] Waiting for the essential requirement 2 of 2: "user session is ready for ssh"
INFO[0045] [hostagent] The essential requirement 1 of 2 is satisfied
INFO[0045] [hostagent] Waiting for the essential requirement 2 of 2: "user session is ready for ssh"
INFO[0045] [hostagent] The essential requirement 2 of 2 is satisfied
INFO[0045] [hostagent] Waiting for the optional requirement 1 of 6: "systemd must be available"
INFO[0045] [hostagent] The essential requirement 2 of 2 is satisfied
INFO[0045] [hostagent] Guest agent is running
INFO[0045] [hostagent] Waiting for the optional requirement 1 of 6: "systemd must be available"
INFO[0045] [hostagent] Not forwarding UDP 127.0.0.54:53
INFO[0045] [hostagent] Not forwarding UDP 127.0.0.53:53
INFO[0045] [hostagent] Guest agent is running
INFO[0045] [hostagent] The optional requirement 1 of 6 is satisfied
INFO[0045] [hostagent] Not forwarding UDP 192.168.105.57:68
INFO[0045] [hostagent] Not forwarding UDP 192.168.5.15:68
INFO[0045] [hostagent] The optional requirement 1 of 6 is satisfied
INFO[0045] [hostagent] Waiting for the optional requirement 2 of 6: "containerd binaries to be installed"
INFO[0045] [hostagent] Not forwarding UDP 127.0.0.54:53
INFO[0045] [hostagent] Not forwarding UDP 127.0.0.53:53
INFO[0045] [hostagent] Waiting for the optional requirement 2 of 6: "containerd binaries to be installed"
INFO[0045] [hostagent] Not forwarding UDP 192.168.105.49:68
INFO[0045] [hostagent] Not forwarding UDP 192.168.5.15:68
INFO[0045] [hostagent] The optional requirement 2 of 6 is satisfied
INFO[0045] [hostagent] Waiting for the optional requirement 3 of 6: "kubeadm to be installed"
INFO[0045] [hostagent] The optional requirement 2 of 6 is satisfied
INFO[0045] [hostagent] Waiting for the optional requirement 3 of 6: "kubeadm to be installed"
INFO[0085] [hostagent] Waiting for the optional requirement 3 of 6: "kubeadm to be installed"
INFO[0085] [hostagent] Waiting for the optional requirement 3 of 6: "kubeadm to be installed"
INFO[0085] [hostagent] The optional requirement 3 of 6 is satisfied
INFO[0085] [hostagent] The optional requirement 3 of 6 is satisfied
INFO[0085] [hostagent] Waiting for the optional requirement 4 of 6: "kubeadm to be completed"
INFO[0085] [hostagent] Waiting for the optional requirement 4 of 6: "kubeadm to be completed"
INFO[0112] [hostagent] The optional requirement 4 of 6 is satisfied
INFO[0112] [hostagent] Waiting for the optional requirement 5 of 6: "kubernetes cluster to be running"
INFO[0118] [hostagent] The optional requirement 5 of 6 is satisfied
INFO[0118] [hostagent] Waiting for the optional requirement 6 of 6: "coredns deployment to be running"
INFO[0124] [hostagent] The optional requirement 4 of 6 is satisfied
INFO[0124] [hostagent] Waiting for the optional requirement 5 of 6: "kubernetes cluster to be running"
INFO[0130] [hostagent] The optional requirement 5 of 6 is satisfied
INFO[0130] [hostagent] Waiting for the optional requirement 6 of 6: "coredns deployment to be running"
INFO[0135] [hostagent] Not forwarding UDP 0.0.0.0:8472
INFO[0147] [hostagent] Not forwarding UDP 0.0.0.0:8472
INFO[0149] [hostagent] The optional requirement 6 of 6 is satisfied
INFO[0149] [hostagent] Waiting for the guest agent to be running
INFO[0149] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0149] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0149] [hostagent] Copying config from /etc/kubernetes/admin.conf to /Users/nsoffer/.lima/c2/copied-from-guest/kubeconfig.yaml
INFO[0149] READY. Run `limactl shell c2` to open the shell.
INFO[0149] Message from the instance "c2":
To run `kubectl` on the host (assumes kubectl is installed), run the following commands:
------
export KUBECONFIG="/Users/nsoffer/.lima/c2/copied-from-guest/kubeconfig.yaml"
kubectl ...
------
INFO[0156] [hostagent] The optional requirement 6 of 6 is satisfied
INFO[0156] [hostagent] Waiting for the guest agent to be running
INFO[0156] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0156] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0156] [hostagent] Copying config from /etc/kubernetes/admin.conf to /Users/nsoffer/.lima/c1/copied-from-guest/kubeconfig.yaml
INFO[0156] READY. Run `limactl shell c1` to open the shell.
INFO[0156] Message from the instance "c1":
To run `kubectl` on the host (assumes kubectl is installed), run the following commands:
------
export KUBECONFIG="/Users/nsoffer/.lima/c1/copied-from-guest/kubeconfig.yaml"
kubectl ...
------
% limactl list
NAME STATUS SSH VMTYPE ARCH CPUS MEMORY DISK DIR
c1 Running 127.0.0.1:62796 vz aarch64 4 4GiB 100GiB ~/.lima/c1
c2 Running 127.0.0.1:62794 vz aarch64 4 4GiB 100GiB ~/.lima/c2
% limactl shell c1 kubectl get node
NAME STATUS ROLES AGE VERSION
lima-c1 Ready control-plane 57s v1.31.1
% limactl shell c2 kubectl get node
NAME STATUS ROLES AGE VERSION
lima-c2 Ready control-plane 77s v1.31.1
% cat ~/.lima/c1/copied-from-guest/kubeconfig.yaml | grep server:
server: https://192.168.105.49:6443
% kubectl get node --kubeconfig ~/.lima/c1/copied-from-guest/kubeconfig.yaml
NAME STATUS ROLES AGE VERSION
lima-c1 Ready control-plane 104s v1.31.1
% cat ~/.lima/c2/copied-from-guest/kubeconfig.yaml | grep server:
server: https://192.168.105.57:6443
% kubectl get node --kubeconfig ~/.lima/c2/copied-from-guest/kubeconfig.yaml
NAME STATUS ROLES AGE VERSION
lima-c2 Ready control-plane 2m3s v1.31.1
AkihiroSuda
commented
1 month ago Thanks, but due to the maintenance cost I'm not sure we want to expand the template list.
Maybe third-party templates can be linked from https://lima-vm.io/docs/templates/
AkihiroSuda
commented
1 month ago Also, I'd suggest to use user-v2 network rather than vmnet for simplicity
nirs
commented
1 month ago Also, I'd suggest to use user-v2 network rather than vmnet for simplicity
We cannot, with user-v2 you cannot access the vms from the host via the ip address, and we don't want to manage port forwarding for multiple clusters, which may also not work with complex setups (e.g connecting clusters with submariner).
nirs
commented
1 month ago Thanks, but due to the maintenance cost I'm not sure we want to expand the template list.
Maybe third-party templates can be linked from https://lima-vm.io/docs/templates/
Make sense, it also more complicated to test, but I would like this to be part of lima and tested with any pr, to ensure that our use case continue work.
AkihiroSuda
commented
1 month ago Make sense, it also more complicated to test, but I would like this to be part of lima and tested with any pr, to ensure that our use case continue work.
Probably the CI of your project should run a canary job with lima@master ?
nirs
commented
1 month ago Make sense, it also more complicated to test, but I would like this to be part of lima and tested with any pr, to ensure that our use case continue work.
Probably the CI of your project should run a canary job with lima@master ?
We use lima only on Apple silicon, and macos-14 does not support nested virtualization, so we cannot run lima vms. Maybe can run lima on x86 runners (macos-13?), I did try this yet.
AkihiroSuda
commented
1 month ago Maybe can run lima on x86 runners (macos-13?), I did try this yet.
Yes, it is what we have been using.
nirs
commented
1 month ago @AkihiroSuda another option is to replace the existing k8s.yaml with this version. It works in the same way, just not depending on port forwarding, and it works for multiple clusters, while the current template does not (only one is accessible via the port forwarding).
nirs
commented
1 month ago Maybe can run lima on x86 runners (macos-13?), I did not try this yet.
Yes, it is what we have been using.
Using nested virtualization or emulation?
AkihiroSuda
commented
1 month ago @AkihiroSuda another option is to replace the existing k8s.yaml with this version. It works in the same way, just not depending on port forwarding, and it works for multiple clusters, while the current template does not (only one is accessible via the port forwarding).
No, k8s.yaml is expected to support port forwarding and should not depend on vmnet.
Probably you can just use k8s.yaml with limactl create --set PROPERTY=VALUE to disable port forwards and enable vmnet.
Using nested virtualization or emulation?
Nested virt.
afbjorklund
commented
1 month ago another option is to replace the existing k8s.yaml with this version
If anything I would like to see k8s.yaml get smaller/simpler, not the opposite...
I don't think that will happen, upstream prefers "README" over fixing get.k8s.io (Otherwise the k8s.yaml could have looked more like the k3s.yaml does today.)
Even flannel is a bit overkill for the single-host scenario, bridge would be just fine.
Had some thoughts on using an external "k8s.sh":
This example show how to run multiple connected kubernetes cluster using a shared network. You can access all the clusters via the shared network, clusters can access each other.
This configuration was derived from k8s.yaml template, with the following changes:
Ensure that the API server and kublet listen on the shared network. This is important for pods that need to listen on the host network. An example is rook ceph test cluster.
Disable port forwarding, since we access the host via the shared network, and automatic port forwarding cannot work for multiple clusters.
Since we access the API server via the shared network, don't modify the kubeconfig copied to the host.
Enable parallel image pulls for faster provisioning of complex clusters.
Allow unprivileged pods to access block devices. Required for kubevirt virtual machines or replicating block volumes using volsync.
These changes were extracted from RamenDR k8s.yaml: https://github.com/RamenDR/ramen/blob/main/test/drenv/providers/lima/k8s.yaml
Based on #2632 since this depends on this change. Without it we need to fixup the route metric during provisioning.