Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

Orygeunik commented 2 years ago

What Happened?

Hello! I'm tried to enable ingress addon, but takes error such as from title partial reference for issue #13841 Originally posted by @spowelljr in https://github.com/kubernetes/minikube/issues/13841#issuecomment-1081007149

root@orion:~# minikube addons enable ingress --alsologtostderr
I0328 21:23:12.286390 1954403 out.go:297] Setting OutFile to fd 1 ...
I0328 21:23:12.286699 1954403 out.go:349] isatty.IsTerminal(1) = true
I0328 21:23:12.286760 1954403 out.go:310] Setting ErrFile to fd 2...
I0328 21:23:12.286819 1954403 out.go:349] isatty.IsTerminal(2) = true
I0328 21:23:12.287050 1954403 root.go:315] Updating PATH: /root/.minikube/bin
W0328 21:23:12.287255 1954403 root.go:293] Error reading config file at /root/.minikube/config/config.json: open /root/.minikube/config/config.json: no such file or directory
I0328 21:23:12.288046 1954403 config.go:176] Loaded profile config "minikube": Driver=none, ContainerRuntime=docker, KubernetesVersion=v1.23.4-rc.0
I0328 21:23:12.288111 1954403 addons.go:65] Setting ingress=true in profile "minikube"
I0328 21:23:12.288159 1954403 addons.go:153] Setting addon ingress=true in "minikube"
I0328 21:23:12.288490 1954403 host.go:66] Checking if "minikube" exists ...
I0328 21:23:12.289084 1954403 exec_runner.go:51] Run: systemctl --version
I0328 21:23:12.294233 1954403 kubeconfig.go:92] found "minikube" server: "https://192.168.188.160:8443"
I0328 21:23:12.294262 1954403 api_server.go:165] Checking apiserver status ...
I0328 21:23:12.294304 1954403 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0328 21:23:12.305745 1954403 exec_runner.go:51] Run: sudo egrep ^[0-9]+:freezer: /proc/1951927/cgroup
W0328 21:23:12.314202 1954403 api_server.go:176] unable to find freezer cgroup: sudo egrep ^[0-9]+:freezer: /proc/1951927/cgroup: exit status 1
stdout:

stderr:
I0328 21:23:12.314232 1954403 api_server.go:240] Checking apiserver healthz at https://192.168.188.160:8443/healthz ...
I0328 21:23:12.319308 1954403 api_server.go:266] https://192.168.188.160:8443/healthz returned 200:
ok
I0328 21:23:12.334410 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
I0328 21:23:12.344282 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0328 21:23:12.354291 1954403 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0328 21:23:12.354596 1954403 addons.go:348] installing /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.354654 1954403 exec_runner.go:144] found /etc/kubernetes/addons/ingress-deploy.yaml, removing ...
I0328 21:23:12.354670 1954403 exec_runner.go:207] rm: /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.354731 1954403 exec_runner.go:151] cp: memory --> /etc/kubernetes/addons/ingress-deploy.yaml (15567 bytes)
I0328 21:23:12.354865 1954403 exec_runner.go:51] Run: sudo cp -a /tmp/minikube3233579437 /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.363407 1954403 exec_runner.go:51] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml
I0328 21:23:12.936710 1954403 addons.go:386] Verifying addon ingress=true in "minikube"
I0328 21:23:12.999722 1954403 out.go:176] 
🔎  Verifying ingress addon...
I0328 21:23:13.000806 1954403 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0328 21:23:13.044867 1954403 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0328 21:23:13.044897 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:13.548198 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:14.048591 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:23:14.547859 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
... cut ...
I0328 21:29:13.050823 1954403 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0328 21:29:13.050850 1954403 kapi.go:108] duration metric: took 6m0.050047714s to wait for app.kubernetes.io/name=ingress-nginx ...
I0328 21:29:13.063110 1954403 out.go:176]

W0328 21:29:13.063269 1954403 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
W0328 21:29:13.063341 1954403 out.go:241]

W0328 21:29:13.072474 1954403 out.go:241]

Attach the log file

logs.txt

Operating System

Other (Debian 11.3)

Driver

none

spowelljr commented 2 years ago

Hi @Orygeunik, thanks for reporting your issue with minikube!

Just looking at the logs I'm seeing bind: address already in use and Operation cannot be fulfilled on jobs.batch "ingress-nginx-admission-create": the object has been modified.

I also see that the cluster was originally created with 1.24.0 and now you're using 1.25.2.

The original cluster might have discrepancies with the updated version, I'd recommend starting with deleting the existing clusters minikube delete --all and then trying starting minikube again and enabling ingress again and let me know if that resolves your issue or your get different errors, thanks!

Orygeunik commented 2 years ago

Thanks for quick answer!

At first i want to remark that this error was received me on version 1.24 and i was hopes that update fixit it :)

So i was executed command minikube delete --all (little question, how to update minikube without errors like this?) and again run command minikube addons enable ingress --alsologtostderr But errror ocured:

W0329 00:26:38.460857 1983999 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

Orygeunik commented 2 years ago

root@orion:~# minikube addons enable ingress --alsologtostderr
I0329 00:20:36.650345 1983999 out.go:297] Setting OutFile to fd 1 ...
I0329 00:20:36.652417 1983999 out.go:349] isatty.IsTerminal(1) = true
I0329 00:20:36.652433 1983999 out.go:310] Setting ErrFile to fd 2...
I0329 00:20:36.652492 1983999 out.go:349] isatty.IsTerminal(2) = true
I0329 00:20:36.652659 1983999 root.go:315] Updating PATH: /root/.minikube/bin
I0329 00:20:36.653128 1983999 config.go:176] Loaded profile config "minikube": Driver=none, ContainerRuntime=docker, KubernetesVersion=v1.23.4-rc.0
I0329 00:20:36.653186 1983999 addons.go:65] Setting ingress=true in profile "minikube"
I0329 00:20:36.653203 1983999 addons.go:153] Setting addon ingress=true in "minikube"
I0329 00:20:36.653407 1983999 host.go:66] Checking if "minikube" exists ...
I0329 00:20:36.653684 1983999 exec_runner.go:51] Run: systemctl --version
I0329 00:20:36.682942 1983999 kubeconfig.go:92] found "minikube" server: "https://192.168.188.160:8443"
I0329 00:20:36.683005 1983999 api_server.go:165] Checking apiserver status ...
I0329 00:20:36.683048 1983999 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0329 00:20:36.730755 1983999 exec_runner.go:51] Run: sudo egrep ^[0-9]+:freezer: /proc/1983293/cgroup
W0329 00:20:36.754976 1983999 api_server.go:176] unable to find freezer cgroup: sudo egrep ^[0-9]+:freezer: /proc/1983293/cgroup: exit status 1
stdout:

stderr:
I0329 00:20:36.755079 1983999 api_server.go:240] Checking apiserver healthz at https://192.168.188.160:8443/healthz ...
I0329 00:20:36.759621 1983999 api_server.go:266] https://192.168.188.160:8443/healthz returned 200:
ok
I0329 00:20:36.770284 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
I0329 00:20:36.779802 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0329 00:20:36.788875 1983999 out.go:176]     ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
I0329 00:20:36.789150 1983999 addons.go:348] installing /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.789188 1983999 exec_runner.go:144] found /etc/kubernetes/addons/ingress-deploy.yaml, removing ...
I0329 00:20:36.789207 1983999 exec_runner.go:207] rm: /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.789274 1983999 exec_runner.go:151] cp: memory --> /etc/kubernetes/addons/ingress-deploy.yaml (15567 bytes)
I0329 00:20:36.789416 1983999 exec_runner.go:51] Run: sudo cp -a /tmp/minikube101386137 /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:36.795590 1983999 exec_runner.go:51] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml
I0329 00:20:38.395316 1983999 exec_runner.go:84] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.23.4-rc.0/kubectl apply -f /etc/kubernetes/addons/ingress-deploy.yaml: (1.599607702s)
I0329 00:20:38.395385 1983999 addons.go:386] Verifying addon ingress=true in "minikube"
I0329 00:20:38.409437 1983999 out.go:176] 🔎  Verifying ingress addon...
🔎  Verifying ingress addon...
I0329 00:20:38.410507 1983999 kapi.go:75] Waiting for pod with label "app.kubernetes.io/name=ingress-nginx" in ns "ingress-nginx" ...
I0329 00:20:38.433520 1983999 kapi.go:86] Found 3 Pods for label selector app.kubernetes.io/name=ingress-nginx
I0329 00:20:38.433657 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:20:38.937061 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:20:39.437494 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
...cut...
I0329 00:26:38.440491 1983999 kapi.go:96] waiting for pod "app.kubernetes.io/name=ingress-nginx", current state: Pending: [<nil>]
I0329 00:26:38.440518 1983999 kapi.go:108] duration metric: took 6m0.030023123s to wait for app.kubernetes.io/name=ingress-nginx ...
I0329 00:26:38.460716 1983999 out.go:176]

W0329 00:26:38.460857 1983999 out.go:241] ❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]
W0329 00:26:38.460873 1983999 out.go:241]

W0329 00:26:38.462117 1983999 out.go:241]

new logs.txt

Orygeunik commented 2 years ago

I'm sorry @spowelljr, anything other need?

spowelljr commented 2 years ago

Hi @Orygeunik, sorry for the delayed response.

After the ingress fails to enable, you can run the command kubectl get pods -A and you should see something like the following:

NAMESPACE       NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8dnv9       0/1     Completed   0          3m14s
ingress-nginx   ingress-nginx-admission-patch-2x7q2        0/1     Completed   1          3m14s
ingress-nginx   ingress-nginx-controller-cc8496874-xxjhc   1/1     Running     0          3m15s
kube-system     coredns-64897985d-zlbf8                    1/1     Running     0          5m3s
kube-system     etcd-minikube                              1/1     Running     0          5m17s
kube-system     kube-apiserver-minikube                    1/1     Running     0          5m15s
kube-system     kube-controller-manager-minikube           1/1     Running     0          5m15s
kube-system     kube-proxy-lzrng                           1/1     Running     0          5m3s
kube-system     kube-scheduler-minikube                    1/1     Running     0          5m15s
kube-system     storage-provisioner                        1/1     Running     0          5m14s

Then if you do kubectl describe pod -n ingress-nginx <pod-name> you should be able to see all of the logs within the pod, I'm assuming one of those ingress pods is going to be failing to start and will have useful errors in there.

Orygeunik commented 2 years ago

Hello @spowelljr

I was rerun again deleting and enabling, such as in comment: https://github.com/kubernetes/minikube/issues/13872#issuecomment-1081169006

And i have below situation:

root@orion:~# kubectl get pods -A
NAMESPACE       NAME                                       READY   STATUS    RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8nlcb       0/1     Pending   0          7m
ingress-nginx   ingress-nginx-admission-patch-jp8wz        0/1     Pending   0          7m
ingress-nginx   ingress-nginx-controller-cc8496874-9lhhr   0/1     Pending   0          7m
kube-system     coredns-64897985d-5hzh5                    0/1     Pending   0          7m18s
kube-system     etcd-orion                                 1/1     Running   6          7m30s
kube-system     kube-apiserver-orion                       1/1     Running   7          7m30s
kube-system     kube-controller-manager-orion              1/1     Running   7          7m30s
kube-system     kube-proxy-pl8nc                           1/1     Running   0          7m19s
kube-system     kube-scheduler-orion                       1/1     Running   6          7m30s
kube-system     storage-provisioner                        0/1     Pending   0          7m24s

kubectl describe pod -n ingress-nginx ingress-nginx-admission-create-8nlcb

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  31s (x390 over 6h40m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

kubectl describe pod -n ingress-nginx ingress-nginx-admission-patch-jp8wz

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  86s (x390 over 6h41m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

kubectl describe pod -n ingress-nginx ingress-nginx-controller-cc8496874-9lhhr

Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

  Warning  FailedScheduling  35s (x396 over 6h45m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

16871560 commented 2 years ago

@Orygeunik do you solve this problem , I got the same issue. oops....

16871560 commented 2 years ago

Orygeunik commented 2 years ago

do you solve this problem , I got the same issue. oops....

@16871560 no, the problem still persisted

16871560 commented 2 years ago

@Orygeunik I reinstall 1.23.2. it seems work well.

Orygeunik commented 2 years ago

@16871560 So, is in my case it no results

root@orion:~# minikube start --driver=none --network-plugin=cni --bootstrapper=kubeadm --kubernetes-version=latest
😄  minikube v1.23.2 on Debian 11.3

root@orion:~# minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

i think so problem around driver, i used "none" and it's a source of problems :(

Orygeunik commented 2 years ago

And so, for tests i was changed driver to "docker" and minikube versio to "v1.23.2" and... it's work

root@orion:~# minikube start --driver=docker --bootstrapper=kubeadm --kubernetes-version=latest --force
😄  minikube v1.23.2 on Debian 11.3

root@orion:~# minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

More proofs:

root@orion:~# minikube start --driver=docker --bootstrapper=kubeadm --kubernetes-version=latest --force
😄  minikube v1.25.2 on Debian 11.3

root@orion:~# minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

culprit is driver - Q.E.D. (and idk why)

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

peachestao commented 2 years ago

@Orygeunik I also encountered this problem, and then check the pod error log, and prompt "0 / 1 nodes are available: 1 node (s) didn't match pod's node affinity / selector Preemption: 0 / 1 nodes are available: 1 preemption is not helpful for scheduling ", i find pod's label "minikube.k8s.io/primary=true" does not exist in the minikube node. later, the problem is solved by running the command:"minikube label node minikube minikube.k8s.io/primary=true"

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

oluranticode commented 1 year ago

Exiting due to MK_ADDON_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

please can anyone help me, I ca't enable ingress on minikube

I'm using mac book Apple M1 chip

klaases commented 1 year ago

please can anyone help me, I ca't enable ingress on minikube

Hi @oluranticode – for additional assistance, please consider reaching out to the minikube community:

https://minikube.sigs.k8s.io/community/

We also offer support through Slack, Groups, and office hours.

manan commented 1 year ago

I have the same issue. It only fails with --driver=none but works otherwise.

techxplrorer commented 1 year ago

Hi @Orygeunik, sorry for the delayed response.

After the ingress fails to enable, you can run the command kubectl get pods -A and you should see something like the following:

NAMESPACE       NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx   ingress-nginx-admission-create-8dnv9       0/1     Completed   0          3m14s
ingress-nginx   ingress-nginx-admission-patch-2x7q2        0/1     Completed   1          3m14s
ingress-nginx   ingress-nginx-controller-cc8496874-xxjhc   1/1     Running     0          3m15s
kube-system     coredns-64897985d-zlbf8                    1/1     Running     0          5m3s
kube-system     etcd-minikube                              1/1     Running     0          5m17s
kube-system     kube-apiserver-minikube                    1/1     Running     0          5m15s
kube-system     kube-controller-manager-minikube           1/1     Running     0          5m15s
kube-system     kube-proxy-lzrng                           1/1     Running     0          5m3s
kube-system     kube-scheduler-minikube                    1/1     Running     0          5m15s
kube-system     storage-provisioner                        1/1     Running     0          5m14s

Then if you do kubectl describe pod -n ingress-nginx <pod-name> you should be able to see all of the logs within the pod, I'm assuming one of those ingress pods is going to be failing to start and will have useful errors in there.

idris@xplorer:~$ kubectl describe pods ingress-nginx-controller-755dfbfc65-s68pr -n ingress-nginx Name: ingress-nginx-controller-755dfbfc65-s68pr Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: minikube/192.168.58.2 Start Time: Wed, 09 Nov 2022 14:24:15 +0800 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx gcp-auth-skip-secret=true pod-template-hash=755dfbfc65 Annotations: Status: Pending IP: 172.17.0.5 IPs: IP: 172.17.0.5 Controlled By: ReplicaSet/ingress-nginx-controller-755dfbfc65 Containers: controller: Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 80/TCP, 443/TCP, 0/TCP Args: /nginx-ingress-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --watch-ingress-without-class=true --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-755dfbfc65-s68pr (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jngmg (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-jngmg: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux minikube.k8s.io/primary=true Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 49m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-755dfbfc65-s68pr to minikube Warning FailedMount 48m (x8 over 49m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found Warning FailedMount 47m kubelet Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert kube-api-access-jngmg]: timed out waiting for the condition Normal Pulling 40m (x3 over 46m) kubelet Pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8" Warning Failed 38m (x3 over 43m) kubelet Error: ErrImagePull Warning Failed 38m (x4 over 43m) kubelet Error: ImagePullBackOff Warning Failed 13m (x8 over 43m) kubelet Failed to pull image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8": rpc error: code = Unknown desc = context deadline exceeded Normal BackOff 4m2s (x101 over 43m) kubelet Back-off pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8"

spowelljr commented 1 year ago

rpc error: code = Unknown desc = context deadline exceeded this is a sign that the operation is taking longer than 2 minutes. The image k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f81 is not huge, but on a machine with slow internet it's possible it's taking longer than 2 minutes to download it.

Could you try running time docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 and see how long it takes to download the image on your machine.

You could also try deleting your minikube cluster (minikube delete) and then start it again with the --extra-config=kubelet.runtime-request-timeout=5m flag which will increase the timeout for the operation to 5 minutes.

techxplrorer commented 1 year ago

@spowelljr Hey! Thanks for your quick response and I followed your steps as below yet it fails to pull the image.

Time taken to pull image

Deleted minikube cluster and increases timeout to 5 mins & 10 mins

Describe Pods

idris@xplorer:~$ kubectl describe pods ingress-nginx-controller-5959f988fd-mqtrp -n ingress-nginx Name: ingress-nginx-controller-5959f988fd-mqtrp Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: minikube/192.168.58.2 Start Time: Fri, 11 Nov 2022 00:57:09 +0800 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx gcp-auth-skip-secret=true pod-template-hash=5959f988fd Annotations: Status: Pending IP: 172.17.0.3 IPs: IP: 172.17.0.3 Controlled By: ReplicaSet/ingress-nginx-controller-5959f988fd Containers: controller: Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 80/TCP, 443/TCP, 0/TCP Args: /nginx-ingress-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-nginx --watch-ingress-without-class=true --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Waiting Reason: ErrImagePull Ready: False Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-5959f988fd-mqtrp (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8qvz9 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-8qvz9: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux minikube.k8s.io/primary=true Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 8m3s default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-5959f988fd-mqtrp to minikube Warning FailedMount 7m31s (x7 over 8m3s) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found Normal Pulling 2m24s (x3 over 6m58s) kubelet Pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8" Warning Failed 25s (x3 over 4m59s) kubelet Failed to pull image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8": rpc error: code = Unknown desc = context deadline exceeded Warning Failed 25s (x3 over 4m59s) kubelet Error: ErrImagePull Normal BackOff 13s (x3 over 4m59s) kubelet Back-off pulling image "k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8" Warning Failed 13s (x3 over 4m59s) kubelet Error: ImagePullBackOff

marcoferri72 commented 1 year ago

As workaround I resolved pre-dowloading problematic image direcly on minikube machine via docker. First connect to minikube container: docker exec -it minikube /bin/bash or simply minikube ssh now download docker image: docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 and then retry to enable ingress addons minikube addons enable ingress

hn-tma commented 1 year ago

As workaround I resolved pre-dowloading problematic image direcly on minikube machine via docker. First connect to minikube container: docker exec -it minikube /bin/bash or simply minikube ssh now download docker image: docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.1@sha256:5516d103a9c2ecc4f026efbd4b40662ce22dc1f824fb129ed121460aaa5c47f8 and then retry to enable ingress addons minikube addons enable ingress

I come here from the similar issue 13841. The aforementioned fix does not work for me. I am using minikube 1.28.0 and kubectl version v1.25.4/v1.25.3 (client/server).

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/minikube/issues/13872#issuecomment-1363631603): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

ClownWang-1217 commented 1 year ago

cause I used minikube, what I got error log like below:

the final solution: minikube delete export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24 minikube start then: minikube addons enable ingress

💡 ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS 💡 After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1" ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.5.1 ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled hope it can help you!

fahimjason commented 1 year ago

cause I used minikube, what I got error log like below:

the final solution: minikube delete export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24 minikube start then: minikube addons enable ingress

bulb ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS bulb After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1" black_small_square Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 black_small_square Using image registry.k8s.io/ingress-nginx/controller:v1.5.1 black_small_square Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 mag_right Verifying ingress addon... star2 The 'ingress' addon is enabled hope it can help you!

Thank you so much @ClownWang-1217. It worked for me.

Jen-apjen commented 1 year ago

Still issue persists. I am using in offline environment in redhat 8.6. Can someone suggest working minikube and kubernetes version?

rhighs commented 1 year ago

The error persists still to this day, I'm using minikube v1.29.0

saphemmy commented 1 year ago

My version of minikube: minikube version minikube version: v1.29.0 This solved it for me: because minikube already had some loaded profiles within its clusters. minikube delete --all minikube start minikube addons enable ingress minikube addons list

fozouni commented 1 year ago

My OS is a Windows and I had the same issue. This is what I've done to get rid of this problem:

1- Run Docker desktop. 2- docker pull nginx/nginx-ingress:latest 3-minikube image load nginx/nginx-ingress:latest 4- minikube addons enable ingress Done! This worked for me.

mnguyen081002 commented 11 months ago

My version of minikube: minikube version minikube version: v1.29.0 This solved it for me: because minikube already had some loaded profiles within its clusters. minikube delete --all minikube start minikube addons enable ingress minikube addons list

This worked for me !

KaiJiaBrother commented 10 months ago

You can try this: https://github.com/kubernetes/minikube/issues/10544#issuecomment-1824087496

rosscreighton commented 2 months ago

cause I used minikube, what I got error log like below:

the final solution: minikube delete export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24 minikube start then: minikube addons enable ingress

💡 ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS 💡 After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1" ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.5.1 ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled hope it can help you!

@ClownWang-1217 why does this work?

Rany-yilian commented 1 month ago

My version of minikube: minikube version minikube version: v1.29.0 This solved it for me: because minikube already had some loaded profiles within its clusters. minikube delete --all minikube start minikube addons enable ingress minikube addons list

this is worked for me,too

kubernetes / minikube