Unable to enable registry add-on.

mprimeaux commented 2 months ago

What Happened?

❯ minikube version
minikube version: v1.33.1
commit: 248d1ec5b3f9be5569977749a725f47b018078ff

minikube config view
- container-runtime: containerd
- cpus: 8
- driver: qemu2
- memory: 16384

minikube start

minikube v1.33.1 on Darwin 14.6.1 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating qemu2 VM (CPUs=8, Memory=16384MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.30.0 on containerd 1.7.15 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
💡  metrics-server is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/metrics-server/metrics-server:v0.7.1
🌟  The 'metrics-server' addon is enabled
vm.max_map_count = 524288
fs.inotify.max_user_instances = 256

minikube addons enable registry

registry is an addon maintained by minikube. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image docker.io/registry:2.8.3
    ▪ Using image gcr.io/k8s-minikube/kube-registry-proxy:0.0.6
🔎  Verifying registry addon...

At this point, the command hangs. Here's what I see in k9s.

Name:             registry-proxy-7c2ml
Namespace:        kube-system
Priority:         0
Service Account:  default
Node:             minikube/192.168.105.9
Start Time:       Wed, 28 Aug 2024 18:57:19 -0500
Labels:           addonmanager.kubernetes.io/mode=Reconcile
                  controller-revision-hash=9784f7967
                  kubernetes.io/minikube-addons=registry
                  pod-template-generation=1
                  registry-proxy=true
Annotations:      <none>
Status:           Pending
IP:               10.244.0.6
IPs:
  IP:           10.244.0.6
Controlled By:  DaemonSet/registry-proxy
Containers:
  registry-proxy:
    Container ID:   
    Image:          gcr.io/k8s-minikube/kube-registry-proxy:0.0.6@sha256:b3fa0b2df8737fdb85ad5918a7e2652527463e357afff83a5e5bb966bcedc367
    Image ID:       
    Port:           80/TCP
    Host Port:      5000/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      REGISTRY_HOST:  registry.kube-system.svc.cluster.local
      REGISTRY_PORT:  80
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qfkfq (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kube-api-access-qfkfq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m36s                default-scheduler  Successfully assigned kube-system/registry-proxy-7c2ml to minikube
  Normal   Pulling    59s (x4 over 2m35s)  kubelet            Pulling image "gcr.io/k8s-minikube/kube-registry-proxy:0.0.6@sha256:b3fa0b2df8737fdb85ad5918a7e2652527463e357afff83a5e5bb966bcedc367"
  Warning  Failed     58s (x4 over 2m33s)  kubelet            Failed to pull image "gcr.io/k8s-minikube/kube-registry-proxy:0.0.6@sha256:b3fa0b2df8737fdb85ad5918a7e2652527463e357afff83a5e5bb966bcedc367": failed to pull and unpack image "gcr.io/k8s-minikube/kube-registry-proxy@sha256:b3fa0b2df8737fdb85ad5918a7e2652527463e357afff83a5e5bb966bcedc367": failed to resolve reference "gcr.io/k8s-minikube/kube-registry-proxy@sha256:b3fa0b2df8737fdb85ad5918a7e2652527463e357afff83a5e5bb966bcedc367": unexpected status from HEAD request to https://gcr.io/v2/k8s-minikube/kube-registry-proxy/manifests/sha256:b3fa0b2df8737fdb85ad5918a7e2652527463e357afff83a5e5bb966bcedc367: 401 Unauthorized
  Warning  Failed     58s (x4 over 2m33s)  kubelet            Error: ErrImagePull
  Warning  Failed     47s (x6 over 2m32s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    35s (x7 over 2m32s)  kubelet            Back-off pulling image "gcr.io/k8s-minikube/kube-registry-proxy:0.0.6@sha256:b3fa0b2df8737fdb85ad5918a7e2652527463e357afff83a5e5bb966bcedc367"

TL;DR is 401 Unauthorized.

What's odd is this has been working for months so perhaps this is an issue on the ghcr.io side of the equation. Insights are welcome.

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

QEMU

ol-teuto commented 2 months ago

Quick and dirty workaround: kubectl get daemonsets.apps -n kube-system registry-proxy -ojson | jq -r '.spec.template.spec.containers[0].image = "gcr.io/google_containers/kube-registry-proxy:0.4"' | kubectl apply -f -

mprimeaux commented 2 months ago

@ol-teuto Thanks much for the reply. Unfortunately, I receive a format error since I'm on ARM64 (M2 Ultra).

exec /bin/boot: exec format error
Stream closed EOF for kube-system/registry-proxy-5c9s9 (registry-proxy)

Based on your work around, the command to enable the registry add-on I used is:

minikube addons enable registry --images='KubeRegistryProxy=gcr.io/google_containers/kube-registry-proxy:0.4

The 0.4 container image doesn't support ARM64.

spowelljr commented 2 months ago

See https://github.com/kubernetes/minikube/issues/19541, this should now be resolved.

kubernetes / minikube