Add Parallels driver support for darwin/arm64

Razorr1996 commented 3 months ago

fixes: #11219

minikube start --driver=parallels # creating new cluster

``` ❯ minikube start --driver=parallels 😄 minikube v1.33.1 on Darwin 14.5 (arm64) ✨ Using the parallels driver based on user configuration 👍 Starting "minikube" primary control-plane node in "minikube" cluster 🔥 Creating parallels VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... 🐳 Preparing Kubernetes v1.30.3 on Docker 27.1.1 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring bridge CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default ```

minikube status

``` ❯ minikube status minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured ```

kubectl get nodes -o wide

``` ❯ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready control-plane 18s v1.30.3 10.211.55.76 Buildroot 2023.02.9 5.10.207 docker://27.1.1 ```

kubectl describe nodes minikube

``` ❯ kubectl describe nodes minikube Name: minikube Roles: control-plane Labels: beta.kubernetes.io/arch=arm64 beta.kubernetes.io/os=linux kubernetes.io/arch=arm64 kubernetes.io/hostname=minikube kubernetes.io/os=linux minikube.k8s.io/commit=732b969929aa87c65157b4450d065c16abacfc41-dirty minikube.k8s.io/name=minikube minikube.k8s.io/primary=true minikube.k8s.io/updated_at=2024_08_04T18_48_25_0700 minikube.k8s.io/version=v1.33.1 node-role.kubernetes.io/control-plane= node.kubernetes.io/exclude-from-external-load-balancers= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sun, 04 Aug 2024 18:48:22 +0200 Taints: Unschedulable: false Lease: HolderIdentity: minikube AcquireTime: RenewTime: Sun, 04 Aug 2024 18:48:44 +0200 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Sun, 04 Aug 2024 18:48:45 +0200 Sun, 04 Aug 2024 18:48:21 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sun, 04 Aug 2024 18:48:45 +0200 Sun, 04 Aug 2024 18:48:21 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sun, 04 Aug 2024 18:48:45 +0200 Sun, 04 Aug 2024 18:48:21 +0200 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Sun, 04 Aug 2024 18:48:45 +0200 Sun, 04 Aug 2024 18:48:28 +0200 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.211.55.76 Hostname: minikube Capacity: cpu: 2 ephemeral-storage: 17734596Ki hugepages-1Gi: 0 hugepages-2Mi: 0 hugepages-32Mi: 0 hugepages-64Ki: 0 memory: 5910208Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 17734596Ki hugepages-1Gi: 0 hugepages-2Mi: 0 hugepages-32Mi: 0 hugepages-64Ki: 0 memory: 5910208Ki pods: 110 System Info: Machine ID: b66cc9e9b275473192aba1fa28ec81ad System UUID: 54cb3b06-1738-4af0-8156-a35887ffe37b Boot ID: 29d14d48-9f25-44c6-b124-6d1dcd6ea811 Kernel Version: 5.10.207 OS Image: Buildroot 2023.02.9 Operating System: linux Architecture: arm64 Container Runtime Version: docker://27.1.1 Kubelet Version: v1.30.3 Kube-Proxy Version: v1.30.3 PodCIDR: 10.244.0.0/24 PodCIDRs: 10.244.0.0/24 Non-terminated Pods: (7 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system coredns-7db6d8ff4d-956br 100m (5%) 0 (0%) 70Mi (1%) 170Mi (2%) 14s kube-system etcd-minikube 100m (5%) 0 (0%) 100Mi (1%) 0 (0%) 28s kube-system kube-apiserver-minikube 250m (12%) 0 (0%) 0 (0%) 0 (0%) 28s kube-system kube-controller-manager-minikube 200m (10%) 0 (0%) 0 (0%) 0 (0%) 28s kube-system kube-proxy-rxgt2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 14s kube-system kube-scheduler-minikube 100m (5%) 0 (0%) 0 (0%) 0 (0%) 29s kube-system storage-provisioner 0 (0%) 0 (0%) 0 (0%) 0 (0%) 27s Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 750m (37%) 0 (0%) memory 170Mi (2%) 170Mi (2%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) hugepages-32Mi 0 (0%) 0 (0%) hugepages-64Ki 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 12s kube-proxy Normal NodeHasSufficientMemory 32s kubelet Node minikube status is now: NodeHasSufficientMemory Normal Starting 28s kubelet Starting kubelet. Normal NodeAllocatableEnforced 28s kubelet Updated Node Allocatable limit across pods Normal NodeHasSufficientMemory 28s kubelet Node minikube status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 28s kubelet Node minikube status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 28s kubelet Node minikube status is now: NodeHasSufficientPID Normal NodeReady 24s kubelet Node minikube status is now: NodeReady Normal RegisteredNode 15s node-controller Node minikube event: Registered Node minikube in Controller ```

minikube --alsologtostderr -v=3 stop # stopping cluster

``` ❯ minikube --alsologtostderr -v=3 stop I0804 18:49:10.157781 74614 out.go:291] Setting OutFile to fd 1 ... I0804 18:49:10.158322 74614 out.go:343] isatty.IsTerminal(1) = true I0804 18:49:10.158330 74614 out.go:304] Setting ErrFile to fd 2... I0804 18:49:10.158335 74614 out.go:343] isatty.IsTerminal(2) = true I0804 18:49:10.158458 74614 root.go:338] Updating PATH: /Users/abasalaev/.minikube/bin I0804 18:49:10.158730 74614 out.go:298] Setting JSON to false I0804 18:49:10.158865 74614 mustload.go:65] Loading cluster: minikube I0804 18:49:10.159066 74614 config.go:182] Loaded profile config "minikube": Driver=parallels, ContainerRuntime=docker, KubernetesVersion=v1.30.3 I0804 18:49:10.159113 74614 profile.go:143] Saving config to /Users/abasalaev/.minikube/profiles/minikube/config.json ... I0804 18:49:10.159463 74614 mustload.go:65] Loading cluster: minikube I0804 18:49:10.159540 74614 config.go:182] Loaded profile config "minikube": Driver=parallels, ContainerRuntime=docker, KubernetesVersion=v1.30.3 I0804 18:49:10.159753 74614 stop.go:39] StopHost: minikube I0804 18:49:10.165336 74614 out.go:177] ✋ Stopping node "minikube" ... ✋ Stopping node "minikube" ... I0804 18:49:10.173187 74614 machine.go:157] backing up vm config to /var/lib/minikube/backup: [/etc/cni /etc/kubernetes] I0804 18:49:10.173282 74614 ssh_runner.go:195] Run: sudo mkdir -p /var/lib/minikube/backup I0804 18:49:10.173528 74614 main.go:141] libmachine: executing: /usr/local/bin/prlctl list minikube --output status --no-header I0804 18:49:10.295991 74614 main.go:141] libmachine: executing: /usr/local/bin/prlctl list -i minikube I0804 18:49:10.482311 74614 main.go:141] libmachine: Found lease: 10.211.55.76 for MAC: 001C42FCF2F8, expiring at 1722791878, leased for 1800 s. I0804 18:49:10.482355 74614 main.go:141] libmachine: Found IP lease: 10.211.55.76 for MAC address 001C42FCF2F8 I0804 18:49:10.482374 74614 sshutil.go:53] new ssh client: &{IP:10.211.55.76 Port:22 SSHKeyPath:/Users/abasalaev/.minikube/machines/minikube/id_rsa Username:docker} I0804 18:49:10.507939 74614 ssh_runner.go:195] Run: sudo rsync --archive --relative /etc/cni /var/lib/minikube/backup I0804 18:49:10.552969 74614 ssh_runner.go:195] Run: sudo rsync --archive --relative /etc/kubernetes /var/lib/minikube/backup I0804 18:49:10.600149 74614 main.go:141] libmachine: Stopping "minikube"... I0804 18:49:10.600192 74614 main.go:141] libmachine: executing: /usr/local/bin/prlctl list minikube --output status --no-header I0804 18:49:10.722496 74614 main.go:141] libmachine: executing: /usr/local/bin/prlctl stop minikube I0804 18:50:09.749292 74614 main.go:141] libmachine: executing: /usr/local/bin/prlctl list minikube --output status --no-header I0804 18:50:10.822492 74614 main.go:141] libmachine: executing: /usr/local/bin/prlctl list minikube --output status --no-header I0804 18:50:10.949578 74614 main.go:141] libmachine: Machine "minikube" was stopped. I0804 18:50:10.949606 74614 stop.go:75] duration metric: took 1m0.775807167s to stop I0804 18:50:10.952477 74614 lock.go:35] WriteFile acquiring /Users/abasalaev/.kube/config: {Name:mk2511360dcfc3b3142e141744412eed433ed661 Clock:{} Delay:500ms Timeout:1m0s Cancel:} I0804 18:50:10.957659 74614 out.go:177] 🛑 1 node stopped. 🛑 1 node stopped. ```

minikube --alsologtostderr -v=3 delete # deleting cluster

``` ❯ minikube --alsologtostderr -v=3 delete I0804 18:51:32.873074 76402 out.go:291] Setting OutFile to fd 1 ... I0804 18:51:32.873454 76402 out.go:343] isatty.IsTerminal(1) = true I0804 18:51:32.873462 76402 out.go:304] Setting ErrFile to fd 2... I0804 18:51:32.873466 76402 out.go:343] isatty.IsTerminal(2) = true I0804 18:51:32.873586 76402 root.go:338] Updating PATH: /Users/abasalaev/.minikube/bin I0804 18:51:32.873854 76402 out.go:298] Setting JSON to false I0804 18:51:32.874039 76402 cli_runner.go:164] Run: docker ps -a --filter label=name.minikube.sigs.k8s.io --format {{.Names}} I0804 18:51:50.601967 76402 cli_runner.go:217] Completed: docker ps -a --filter label=name.minikube.sigs.k8s.io --format {{.Names}}: (17.72766875s) I0804 18:51:50.605109 76402 config.go:182] Loaded profile config "bridge-193000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.30.2 I0804 18:51:50.605414 76402 config.go:182] Loaded profile config "flannel-193000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.30.2 I0804 18:51:50.605530 76402 config.go:182] Loaded profile config "minikube": Driver=parallels, ContainerRuntime=docker, KubernetesVersion=v1.30.3 I0804 18:51:50.605664 76402 config.go:182] Loaded profile config "minikube": Driver=parallels, ContainerRuntime=docker, KubernetesVersion=v1.30.3 I0804 18:51:50.605691 76402 delete.go:301] DeleteProfiles I0804 18:51:50.605702 76402 delete.go:329] Deleting minikube I0804 18:51:50.605713 76402 delete.go:334] minikube configuration: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube-builds/iso/19339/minikube-v1.33.1-1722248113-19339-arm64.iso KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:6000 CPUs:2 DiskSize:20000 Driver:parallels HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.3 ClusterName:minikube Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:10.211.55.76 Port:8443 KubernetesVersion:v1.30.3 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[default-storageclass:true storage-provisioner:true] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} I0804 18:51:50.605900 76402 config.go:182] Loaded profile config "minikube": Driver=parallels, ContainerRuntime=docker, KubernetesVersion=v1.30.3 I0804 18:51:50.606229 76402 config.go:182] Loaded profile config "minikube": Driver=parallels, ContainerRuntime=docker, KubernetesVersion=v1.30.3 I0804 18:51:50.607063 76402 main.go:141] libmachine: executing: /usr/local/bin/prlctl list minikube --output status --no-header I0804 18:51:50.738839 76402 out.go:177] 🔥 Deleting "minikube" in parallels ... 🔥 Deleting "minikube" in parallels ... I0804 18:51:50.744500 76402 main.go:141] libmachine: executing: /usr/local/bin/prlctl list minikube --output status --no-header I0804 18:51:50.886207 76402 main.go:141] libmachine: executing: /usr/local/bin/prlctl stop minikube --kill I0804 18:51:52.233598 76402 main.go:141] libmachine: executing: /usr/local/bin/prlctl delete minikube I0804 18:51:52.371577 76402 lock.go:35] WriteFile acquiring /Users/abasalaev/.kube/config: {Name:mk2511360dcfc3b3142e141744412eed433ed661 Clock:{} Delay:500ms Timeout:1m0s Cancel:} I0804 18:51:52.379028 76402 out.go:177] 💀 Removed all traces of the "minikube" cluster. 💀 Removed all traces of the "minikube" cluster. ```

k8s-ci-robot commented 3 months ago

Hi @Razorr1996. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

minikube-bot commented 3 months ago

Can one of the admins verify this patch?

Razorr1996 commented 3 months ago

/assign

Razorr1996 commented 3 months ago

/cc @afbjorklund

medyagh commented 3 months ago

/ok-to-test

minikube-pr-bot commented 3 months ago

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19373) |
+----------------+----------+---------------------+
| minikube start | 54.4s    | 54.1s               |
| enable ingress | 27.3s    | 27.3s               |
+----------------+----------+---------------------+

Times for minikube ingress: 29.8s 27.6s 27.7s 23.7s 27.7s Times for minikube (PR 19373) ingress: 27.1s 26.2s 27.6s 29.3s 26.2s Times for minikube start: 51.8s 56.0s 53.3s 56.7s 54.2s Times for minikube (PR 19373) start: 52.5s 56.6s 54.5s 54.0s 53.0s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19373) |
+----------------+----------+---------------------+
| minikube start | 24.8s    | 24.5s               |
| enable ingress | 21.9s    | 21.8s               |
+----------------+----------+---------------------+

Times for minikube ingress: 22.4s 21.4s 21.9s 21.9s 21.9s Times for minikube (PR 19373) ingress: 21.9s 21.9s 21.9s 21.9s 21.4s Times for minikube start: 26.3s 23.1s 25.3s 23.6s 25.6s Times for minikube (PR 19373) start: 25.6s 27.2s 22.8s 23.0s 23.7s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19373) |
+----------------+----------+---------------------+
| minikube start | 24.3s    | 22.7s               |
| enable ingress | 47.9s    | 40.6s               |
+----------------+----------+---------------------+

Times for minikube start: 25.3s 23.2s 24.7s 22.4s 26.0s Times for minikube (PR 19373) start: 25.0s 22.0s 21.0s 22.5s 22.8s Times for minikube ingress: 48.9s 47.9s 46.9s 46.9s 48.9s Times for minikube (PR 19373) ingress: 40.1s 31.9s 48.9s 47.9s 34.0s

medyagh commented 3 months ago

/lgtm

k8s-ci-robot commented 3 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, Razorr1996

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes/minikube/blob/master/OWNERS)~~ [medyagh] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

minikube-pr-bot commented 3 months ago

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Linux_crio_arm64 (5 failed)	TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup(gopogh)	0.00% (chart)
Docker_Linux_crio_arm64 (5 failed)	TestFunctional/parallel/TunnelCmd/serial/AccessDirect(gopogh)	0.00% (chart)
Docker_Linux_crio_arm64 (5 failed)	TestFunctional/parallel/PersistentVolumeClaim(gopogh)	2.37% (chart)
Docker_Linux_containerd_arm64 (2 failed)	TestStartStop/group/old-k8s-version/serial/SecondStart(gopogh)	46.75% (chart)

Besides the following environments also have failed tests:

Docker_Linux_crio: 2 failed (gopogh)
KVM_Linux_crio: 14 failed (gopogh)
QEMU_macOS: 94 failed (gopogh)
Hyperkit_macOS: 26 failed (gopogh)
Docker_Cloud_Shell: 5 failed (gopogh)

To see the flake rates of all tests by environment, click here.

kubernetes / minikube

Add Parallels driver support for darwin/arm64 #19373