Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

[dgoldssfo]

Note - I have this problem with 1.18.1 and 1.19.0 Is there a version I can back off to that will work?

Steps to reproduce the issue:

1. forgeops git:(master) ✗ minikube start --memory=10240 --cpus=3 --disk-size=40g --cni=true --vm=true \ --driver=virtualbox --bootstrapper kubeadm --kubernetes-version=1.20.0
2. minikube addons enable ingress

Full output of minikube start command used, if not already included: (also includes o/p from minikube addons enable ingress)

➜ forgeops git:(master) ✗ minikube start --memory=10240 --cpus=3 --disk-size=40g --cni=true --vm=true \ --driver=virtualbox --bootstrapper kubeadm --kubernetes-version=1.20.0 😄 minikube v1.19.0 on Darwin 11.2.2 ✨ Using the virtualbox driver based on user configuration 💿 Downloading VM boot image ...

minikube-v1.19.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s minikube-v1.19.0.iso: 244.49 MiB / 244.49 MiB 100.00% 5.94 MiB p/s 41.34 👍 Starting control plane node minikube in cluster minikube 💾 Downloading Kubernetes v1.20.0 preload ... preloaded-images-k8s-v10-v1...: 491.68 MiB / 491.68 MiB 100.00% 5.63 MiB 🔥 Creating virtualbox VM (CPUs=3, Memory=10240MB, Disk=40960MB) ... 🐳 Preparing Kubernetes v1.20.0 on Docker 20.10.4 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring CNI (Container Networking Interface) ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default ➜ forgeops git:(master) ✗ minikube addons enable ingress
▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0 ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1 ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1 🔎 Verifying ingress addon...

❌ Exiting due to MK_ENABLE: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: timed out waiting for the condition]

I tried to pull the Docker file that was causing the problem; stuck on waiting:

➜ forgeops git:(master) ✗ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx ingress-nginx-admission-create-dhp4x 0/1 Completed 0 86m ingress-nginx ingress-nginx-admission-patch-h74ws 0/1 Completed 0 86m ingress-nginx ingress-nginx-controller-5d88495688-2bgln 0/1 ImagePullBackOff 0 86m kube-system coredns-74ff55c5b-lsmmx 1/1 Running 0 87m kube-system etcd-minikube 1/1 Running 0 87m kube-system kindnet-cq5xj 1/1 Running 0 87m kube-system kube-apiserver-minikube 1/1 Running 0 87m kube-system kube-controller-manager-minikube 1/1 Running 0 87m kube-system kube-proxy-zrscz 1/1 Running 0 87m kube-system kube-scheduler-minikube 1/1 Running 0 87m kube-system storage-provisioner 1/1 Running 0 87m ➜ forgeops git:(master) ✗ kubectl describe pod/ingress-nginx-controller-5d88495688-2bgln --namespace=kube-system Error from server (NotFound): pods "ingress-nginx-controller-5d88495688-2bgln" not found ➜ forgeops git:(master) ✗ kubectl describe pod/ingress-nginx-controller-5d88495688-2bgln --namespace=ingress-nginx Name: ingress-nginx-controller-5d88495688-2bgln Namespace: ingress-nginx Priority: 0 Node: minikube/192.168.99.100 Start Time: Mon, 19 Apr 2021 11:58:25 -0700 Labels: addonmanager.kubernetes.io/mode=Reconcile app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx gcp-auth-skip-secret=true pod-template-hash=5d88495688 Annotations: Status: Pending IP: 10.244.0.5 IPs: IP: 10.244.0.5 Controlled By: ReplicaSet/ingress-nginx-controller-5d88495688 Containers: controller: Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 80/TCP, 443/TCP, 0/TCP Args: /nginx-ingress-controller --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --report-node-internal-ip-address --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services --udp-services-configmap=$(POD_NAMESPACE)/udp-services --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-5d88495688-2bgln (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-qzbqr (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false ingress-nginx-token-qzbqr: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-token-qzbqr Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Warning Failed 41m (x11 over 84m) kubelet Error: ErrImagePull Normal Pulling 17m (x15 over 87m) kubelet Pulling image "k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a" Warning Failed 7m6s (x263 over 84m) kubelet Error: ImagePullBackOff Normal BackOff 2m10s (x280 over 84m) kubelet Back-off pulling image "k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a" ➜ forgeops git:(master) ✗ minikube ssh
( ) ( )
(_) ()| |/') | |_
/' ` \| |/' _| || , < ( ) ( )| '_`\ /'\ | ( ) ( ) || || ( ) || || |\\ | () || |) )( _/ () () ()()() ()()() (_)\___/'(_,__/'____)

$ docker pull k8s.gcr.io/ingress-nginx/controller:v0.44.0@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a k8s.gcr.io/ingress-nginx/controller@sha256:3dd0fac48073beaca2d67a78c746c7593f9c575168a17139a9955a82c63c4b9a: Pulling from ingress-nginx/controller 596ba82af5aa: Already exists b198ba082145: Already exists c7166c196d5e: Already exists addc1fbd9812: Already exists f9949b841da7: Already exists 97335954f799: Waiting 826bbedb287b: Waiting

[dgoldssfo]

Stuck on "Waiting" for at least ten minutes now.

[dgoldssfo]

A friend suggested I blow away my ~/.minikube directory and try it, which solved the problem.

If you don't want to investigate why having cruft in that directory caused this problem, go ahead and close this. But it might be good to look into it. Your call, thanks for all the great work you folks do.

[spowelljr]

@dgoldssfo Thanks for the info, did you update from an older version of minikube previous to this?

[dgoldssfo]

Yes, I updated from 1.18.1 to 1.19.0.

But note - enabling the ingress addon had gotten very slow (like, it took several minutes). After deleting .minikube, it was back to taking < 10 seconds.

[spowelljr]

Thank you for you info, you're not the only person who experienced this issue. I'll see if I can replicate the same issue.

[dgoldssfo]

I'm also having other ingress issues. Investigating now.

[dgoldssfo]

Could be our app, but maybe this'll help:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding

[dgoldssfo]

Looks like it's the ingress controller plugin. Is there a new one in 1.19?

Trying to ctreate this ingress fails, for example:

---

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: cert-manager.io/cluster-issuer: default-issuer kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/configuration-snippet: | rewrite ^(/platform)$ $1/ redirect; nginx.ingress.kubernetes.io/rewrite-target: /$1 labels: app.kubernetes.io/part-of: forgerock name: platform-ui spec: rules:

• host: default.iam.example.com http: paths:
  • backend: service: name: admin-ui port: number: 8080 path: /platform/?(.*) pathType: Prefix tls:
• hosts:
  • default.iam.example.com secretName: sslcert

[medyagh]

thank you everyone for reporting this bug, this seems to be side effect of this PR #10879 that we tried to update the ingress to new version, unfortunately our integration test did not catch this problem, Please stay tuned, we will create a fix for this and hopefully by adding an integration test this should never happen again.

meanwhile could you try

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

and see if that fixes the problem ?

I suspect this is dupe of https://github.com/kubernetes/minikube/issues/11121

[vgryzunov]

Tried kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission. Did not help. Had to re-create minikube by

minikube delete --all --purge
minikube addons enable ingress

[spowelljr]

Hi @dgoldssfo, thanks for reporting your issue!

This is a duplicate of #10544, in order to keep all discussion of this issue centralized I'm going to close this issue.

Please follow along with #10544 and contribute any findings there, thanks for your understanding!

[gilkremer123]

Had the same issue (on windows). could not enable ingress. turns out running CMD as an administrator solved it :) So it was just a permissions issue

[Monte-Christo]

Had the same issue on Win 10 Enterprise. When I switched from Admin PowerShell to Admin Cmd Shell, the command completed without any fuss. Go figure.

[Richard-code-gig]

In Admin CMD this should resolve it: minikube delete --all --purge minikube start minikube addons enable ingress