MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found

[kundan2707]

NGINX Ingress controller version:0.34.1

Kubernetes version (use kubectl version):1.18.1

Environment:

Cloud provider or hardware configuration:baremetal OS (e.g. from /etc/os-release):ubuntu16.04 Kernel (e.g. uname -a):4.4.0-184-generic.x86_64 Install tools:deploy.yaml

What happened: ingress-nginx-admission-create and ingress-nginx-admission-patch started intially but Ingress-nginx controller failed to start

NAMESPACE        NAME                                        READY   STATUS              RESTARTS   AGE
ingress-nginx    ingress-nginx-admission-create-6sph6        1/1     Running             2          86s
ingress-nginx    ingress-nginx-admission-patch-ks2cm         1/1     Running             2          86s
ingress-nginx    ingress-nginx-controller-7fd7d8df56-wpdnp   0/1     ContainerCreating   0          96s

What you expected to happen: Ingress-nginx controller should start successfully

How to reproduce it:

Install kubernetes cluster using kubeadm

Install the ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

Anything else we need to know:

vagrant@k8s-head:~$ kubectl describe pod ingress-nginx-controller-7fd7d8df56-wpdnp -n ingress-nginx
Name:           ingress-nginx-controller-7fd7d8df56-wpdnp
Namespace:      ingress-nginx
Priority:       0
Node:           k8s-node-1/192.168.205.11
Start Time:     Fri, 24 Jul 2020 01:49:42 +0000
Labels:         app.kubernetes.io/component=controller
                app.kubernetes.io/instance=ingress-nginx
                app.kubernetes.io/name=ingress-nginx
                pod-template-hash=7fd7d8df56
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/ingress-nginx-controller-7fd7d8df56
Containers:
  controller:
    Container ID:  
    Image:         us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1@sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20
    Image ID:      
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=ingress-nginx/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7fd7d8df56-wpdnp (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-4t6nq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  ingress-nginx-token-4t6nq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-token-4t6nq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age               From                 Message
  ----     ------       ----              ----                 -------
  Normal   Scheduled    2m1s              default-scheduler    Successfully assigned ingress-nginx/ingress-nginx-controller-7fd7d8df56-wpdnp to k8s-node-1
  Warning  FailedMount  57s (x8 over 2m)  kubelet, k8s-node-1  MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found

[roy-work]

I'm also hitting this problem.

The secret is created after these two jobs finish. After that, the pod will start.

The two jobs mentioned … don't exist. So that "explains" why things that depend on them don't start. The output from helm indicates that they never templated.

Now, they start with:

{{- if and .Values.controller.admissionWebhooks.enabled .Values.controller.admissionWebhooks.patch.enabled -}}

But that condition should evaluate to true. (I'm using the defaults, and those are enabled.)

A later version of Helm generates the right output (that file renders), so this seems to be a combination of Helm & this chart. (But I'm not in a position to upgrade Helm, here…)

Edit: I am hitting this bug in Helm.

Edit edit: And conflict! I remembered why we're on the version of Helm we're on: we can't upgrade Helm as later changes introduced this bug which breaks other charts we have, and that bug is present in versions ≥3.2 (and remains unfixed, don't let the link to a closed issue fool you!). There is literally no version of Helm that works.

ingress-nginx could side-step this by flattening their directory structure… but IDK. ingress-nginx doesn't appear to be in the wrong, here.

Edit edit edit: Nope, even if I flatten a/b/c.yaml to a--b--c.yaml I still get the wrong render. I have no idea anymore what's going on.

Edit⁴: It's https://github.com/helm/helm/issues/7001. Helm's got more holes than swiss cheese. Helm hooks are a bit interesting ; we don't use helm, rather, we use Flux, and I'm not sure what its story is for hooks.

[longwuyuan]

There are static yaml manifests published by the project. The docs says so

[simon-wessel]

I moved the controller from the default namespace to a dedicated ingress namespace and ran into this issue. Deleting the existing admission secret in the old default namespace and then redeploying the controller solved my issue.

[SaberStrat]

To whoever might stumble over this issue:

In my case, this same Event's cause turned out to be the ingress-nginx-admission-create Job's Pod not being able to pull its image due to a faulty Proxy setting.

[srt180]

国内的朋友把镜像源替换一下再试试

[hamza528]

you should use a cri-o container runtime not docker or containerd its worked for me

[Josh-a-e]

In my case, this issue was related to namespaces. It came down to the difference between

(Pulumi based)

❌ Resulted in the discussed issue

  const ingressNamespace = new kubernetes.core.v1.Namespace(
    'k8s-mast-namespace-ingress-nginx',
    { metadata: { name: 'ingress-nginx' } }
  );
  // https://www.pulumi.com/registry/packages/kubernetes/api-docs/helm/v3/release/
  // https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-on-digitalocean-kubernetes-using-helm
  new kubernetes.helm.v3.Release(
    'k8s-mast-helm-chart-ingress-nginx',
    {
      chart: 'ingress-nginx',
      // versions https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx
      version: '4.7.2',
      repositoryOpts: {
        repo: 'https://kubernetes.github.io/ingress-nginx',
      },
      namespace: ingressNamespace.metadata.name,
      values: {
        controller: {
          publishService: { enabled: true },
          metrics: { enabled: true },
        },
      },
    }
  );

✅ Worked

  new kubernetes.helm.v3.Release(
    'k8s-mast-helm-chart-ingress-nginx',
    {
      chart: 'ingress-nginx',
      // versions https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx
      version: '4.7.2',
      repositoryOpts: {
        repo: 'https://kubernetes.github.io/ingress-nginx',
      },
      namespace: 'ingress-nginx',
      createNamespace: true, // ⬅️ critical change
      values: {
        controller: {
          publishService: { enabled: true },
          metrics: { enabled: true },
        },
      },
    }
  );