search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.16k stars 4.87k forks source link

Minikube Ingress Container Pull Error #14613

Closed nwhln-sgrid closed 1 year ago

nwhln-sgrid commented 2 years ago

What Happened?

I cannot pull the ingress container. My current minikube version:

USER@HOSTNAME:~/awx-prod$ minikube version
minikube version: v1.26.0
commit: f4b412861bb746be73053c9f6d2895f12cf78565

Prior to attempting firing up minikube:

USER@HOSTNAME:~/awx-prod$ minikube stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.
USER@HOSTNAME:~/awx-prod$ minikube delete --all
🔥  Deleting "minikube" in docker ...
🔥  Removing /home/nwhalen/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles

Attempting to start minikube with ingress:

USER@HOSTNAME:~/awx-prod$ minikube start --addons=ingress --cpus=2 --cni=flannel --install-addons=true --kubernetes-version=stable --memory=6g --driver docker
😄  minikube v1.26.0 on Ubuntu 20.04
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=6144MB) ...
🐳  Preparing Kubernetes v1.24.1 on Docker 20.10.17 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Flannel (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying ingress addon...
❗  Enabling 'ingress' returned an error: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Attach the log file

minikube.log

Operating System

Ubuntu

Driver

Docker

ashok-arora commented 2 years ago

Is it related to https://github.com/kubernetes/minikube/issues/8756?

nwhln-sgrid commented 2 years ago

Is it related to #8756?

It appears the two are similar or identical.

nwhln-sgrid commented 2 years ago

When I check the kubectl deployment via kubectl -n ingress-nginx edit deployment ingress-nginx-controller The pod image URL is listed as: k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 Which returns a 404.

nwhln-sgrid commented 2 years ago

When I attempt to pull via docker: 

USER@HOSTNAME:~$ docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1
v1.2.1: Pulling from ingress-nginx/controller
8663204ce13b: Pull complete 
897a18b2d257: Pull complete 
3cb02f360cf3: Pull complete 
2b63816a7692: Pull complete 
d61ce16aa3b6: Pull complete 
4391833fbf2c: Pull complete 
4f4fb700ef54: Pull complete 
bb397308bcd5: Pull complete 
803395581751: Pull complete 
153d402a7263: Pull complete 
c815f058cf7b: Pull complete 
a872540e4aca: Pull complete 
4972574251d0: Pull complete 
30197fe775a6: Pull complete 
b059831ea274: Pull complete 
Digest: sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
Status: Downloaded newer image for k8s.gcr.io/ingress-nginx/controller:v1.2.1
k8s.gcr.io/ingress-nginx/controller:v1.2.1
nwhln-sgrid commented 2 years ago

For some reason, after installing virtualbox & virtualbox-dkms, the ingress add-on works. I stopped minikube afterwards, deleted the cluster, rebooted the server & re-ran the minikube command successfully:

USER@HOSTNAME:~$ minikube start --addons=ingress --cpus=2 --cni=flannel --install-addons=true --kubernetes-version=stable --memory=6g --driver=virtualbox
😄  minikube v1.26.0 on Ubuntu 20.04
✨  Using the virtualbox driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating virtualbox VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.24.1 on Docker 20.10.16 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Flannel (Container Networking Interface) ...
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying ingress addon...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner, ingress
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
nwhln-sgrid commented 2 years ago

Some more background on the issue, it appears to be due to a missing secret: kubectl describe pod ingress-nginx-controller-755dfbfc65-52zqr -n ingress-nginx

Name:           ingress-nginx-controller-755dfbfc65-52zqr
Namespace:      ingress-nginx
Priority:       0
Node:           minikube/192.168.49.2
Start Time:     Thu, 21 Jul 2022 20:26:19 +0000
Labels:         app.kubernetes.io/component=controller
                app.kubernetes.io/instance=ingress-nginx
                app.kubernetes.io/name=ingress-nginx
                gcp-auth-skip-secret=true
                pod-template-hash=755dfbfc65
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/ingress-nginx-controller-755dfbfc65
Containers:
  controller:
    Container ID:  
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8
    Image ID:      
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-755dfbfc65-52zqr (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kmrg2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-kmrg2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    6m12s                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-755dfbfc65-52zqr to minikube
  Warning  FailedMount  4m9s                   kubelet            Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-kmrg2]: timed out waiting for the condition
  Warning  FailedMount  2m1s (x10 over 6m11s)  kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Warning  FailedMount  114s                   kubelet            Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[kube-api-access-kmrg2 webhook-cert]: timed out waiting for the condition

kubectl get secret -n ingress-nginx No resources found in ingress-nginx namespace.

nwhln-sgrid commented 2 years ago

Anyone have any ideas?

RA489 commented 2 years ago

/kind support

rhythmshandlya commented 2 years ago

image Is this the same issue? I am stuck here, containers are not ready.

dgoldssfo commented 2 years ago

I'm experiencing the same problem with minikube 1.26.0. minikube start takes around 10 minutes and then times out about half the time.

What I am seeing:

Configuring CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 ▪ Using image k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1 ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 🔎 Verifying ingress addon... ❗ Enabling 'ingress' returned an error: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition] 🌟 Enabled addons: storage-provisioner, default-storageclass, volumesnapshots 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

When I run kubectl get pods -n ingress-nginx while minikube start is executing, I observe that the ingress-nginx-admission-create and ingress-nginx-admission-patch take maybe 5 minutes to run, and the ingress controller pod is in ContainerCreating state while that's going on. Eventually, the create/patch jobs finish, and sooner or later, the ingress-controller pod becomes ready. Sometimes it becomes ready before the timeout, sometimes after.

If the timeout happens first, I can enable the addon with a separate command, and it's successful:

➜ forgeops git:(master) minikube addons enable ingress ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1 ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled

(Does anyone know if I need to run this separate step to enable the addon or not, or does "enable addon" just wait to see that the ingress controller pod is active?)

klaases commented 1 year ago

(Does anyone know if I need to run this separate step to enable the addon or not, or does "enable addon" just wait to see that the ingress controller pod is active?)

Hi @dgoldssfo – is this issue still occurring? Were you able to find a solution?

For additional assistance, please consider reaching out to the minikube community:

https://minikube.sigs.k8s.io/community/

We also offer support through Slack, Groups, and office hours.

dgoldssfo commented 1 year ago

Seems like it's been fixed. minikube start is back to about what it used to take, and I haven't seen a timeout in a while.

klaases commented 1 year ago

Seems like it's been fixed. minikube start is back to about what it used to take, and I haven't seen a timeout in a while.

Hi @dgoldssfo, thanks for the follow up response to the issue.

If anything else comes up please feel free to re-open the issue by commenting with /reopen.

Thank you for sharing your experience!

ongiant commented 1 year ago

(Does anyone know if I need to run this separate step to enable the addon or not, or does "enable addon" just wait to see that the ingress controller pod is active?)

Hi @dgoldssfo – is this issue still occurring? Were you able to find a solution?

For additional assistance, please consider reaching out to the minikube community:

https://minikube.sigs.k8s.io/community/

We also offer support through Slack, Groups, and office hours.

I joined unsuccessfully and was prompted that I don't have a @get-an-invite-at.slack.kubernetes.io email. What should I do? Where can I contact the administrator of k8s workspace?

hedefalk commented 1 year ago

Same here.

minikube addons enable ingress fails:

🐟 minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
💡  After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.7.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

If I look at the pods:

✗ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS              RESTARTS   AGE
ingress-nginx-admission-create-kcj4q        0/1     ImagePullBackOff    0          7m5s
ingress-nginx-admission-patch-pw5dk         0/1     ImagePullBackOff    0          7m5s
ingress-nginx-controller-6cc5ccb977-zf2kr   0/1     ContainerCreating   0          7m5s

I can see that there's a pull error:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  24m                   default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-create-kcj4q to minikube
  Normal   Pulling    23m (x4 over 24m)     kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"
  Warning  Failed     23m (x4 over 24m)     kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": rpc error: code = Unknown desc = Error response from daemon: Get "https://europe-north1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": dial tcp: lookup europe-north1-docker.pkg.dev on 192.168.65.254:53: no such host
  Warning  Failed     23m (x4 over 24m)     kubelet            Error: ErrImagePull
  Warning  Failed     23m (x6 over 24m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m44s (x88 over 24m)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f"'

And if I go into minikube and manually pull:

ssh minikube

docker@minikube:~$ docker pull registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f
Error response from daemon: Get "https://europe-north1-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f": dial tcp: lookup europe-north1-docker.pkg.dev on 192.168.65.254:53: no such host

However, if I just replace the domain:

docker@minikube:~$ docker pull k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v20230312-helm-chart-4.5.2-28-g66a760794@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f
k8s.gcr.io/ingress-nginx/kube-webhook-certgen@sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f: Pulling from ingress-nginx/kube-webhook-certgen
Digest: sha256:01d181618f270f2a96c04006f33b2699ad3ccb02da48d0f89b22abce084b292f

…all works fine.

hedefalk commented 1 year ago

registry.k8s.io redirects to europe-north1-docker.pkg.dev that doesn't resolve on minikube?

docker@minikube:~$ nslookup europe-north1-docker.pkg.dev
Server:     192.168.65.254
Address:    192.168.65.254#53

** server can't find europe-north1-docker.pkg.dev: NXDOMAIN

docker@minikube:~$ nslookup registry.k8s.io             
Server:     192.168.65.254
Address:    192.168.65.254#53

Non-authoritative answer:
Name:   registry.k8s.io
Address: 34.96.108.209
Name:   registry.k8s.io
Address: 2600:1901:0:bbc4::

docker@minikube:~$ nslookup k8s.gcr.io     
Server:     192.168.65.254
Address:    192.168.65.254#53

Non-authoritative answer:
k8s.gcr.io  canonical name = googlecode.l.googleusercontent.com.
Name:   googlecode.l.googleusercontent.com
Address: 173.194.222.82
Name:   googlecode.l.googleusercontent.com
Address: 2a00:1450:4010:c0b::52