nginx-ingress pod fails to launch with kind cluster

[psankar]

What happened:

I created a kind cluster as follows:

❯ cat kind-with-registry.sh 
#!/bin/sh

# Copied from https://kind.sigs.k8s.io/docs/user/local-registry/

set -o errexit

# create registry container unless it already exists
reg_name='kind-registry'
reg_port='5001'
if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then
  docker run \
    -d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}" \
    registry:2
fi

# create a cluster with the local registry enabled in containerd
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
    endpoint = ["http://${reg_name}:5000"]
nodes:
- role: control-plane
- role: worker
- role: worker
EOF

# connect the registry to the cluster network if not already connected
if [ "$(docker inspect -f='{{json .NetworkSettings.Networks.kind}}' "${reg_name}")" = 'null' ]; then
  docker network connect "kind" "${reg_name}"
fi

# Document the local registry
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "localhost:${reg_port}"
    help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF

It has a control-plane node and a couple of worker nodes. Then I tried creating an ingress via nginx.

$ kubectl get nodes 
NAME                 STATUS   ROLES           AGE     VERSION
kind-control-plane   Ready    control-plane   2m19s   v1.25.3
kind-worker          Ready    <none>          114s    v1.25.3
kind-worker2         Ready    <none>          114s    v1.25.3

$ kubectl apply -f https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/kind/deploy.yaml

However, the nginx ingress controller pod does not come up and remains in the Pending state forever, complaining about Node taints

❯ kubectl describe pod -n ingress-nginx ingress-nginx-controller-6bccc5966-tgn2f 
Name:             ingress-nginx-controller-6bccc5966-tgn2f
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             <none>
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  pod-template-hash=6bccc5966
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/ingress-nginx-controller-6bccc5966
Containers:
  controller:
    Image:       registry.k8s.io/ingress-nginx/controller:v1.5.1@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629
    Ports:       80/TCP, 443/TCP, 8443/TCP
    Host Ports:  80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --watch-ingress-without-class=true
      --publish-status-address=localhost
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6bccc5966-tgn2f (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-czk2z (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-czk2z:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              ingress-ready=true
                             kubernetes.io/os=linux
Tolerations:                 node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m14s  default-scheduler  0/3 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

It is likely that there is no bug in kind or docker and I may just need to do some extra steps to manage the taints. But the documentation at https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx does not refer to any extra steps.

Environment:

• kind version: (use kind version):

  
  ❯ kind version
  kind v0.17.0 go1.19.3 linux/amd64

- Runtime info: (use `docker info` or `podman info`):

❯ docker info Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Docker Buildx (Docker Inc., v0.9.1-docker) compose: Docker Compose (Docker Inc., v2.12.2) scan: Docker Scan (Docker Inc., v0.21.0)

Server: Containers: 6 Running: 4 Paused: 0 Stopped: 2 Images: 79 Server Version: 20.10.21 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 1c90a442489720eec95342e1789ee8a5e1b9536f runc version: v1.1.4-0-g5fd4c4d init version: de40ad0 Security Options: apparmor seccomp Profile: default cgroupns Kernel Version: 5.15.0-56-generic Operating System: Ubuntu 22.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.29GiB Name: psankar ID: GG7E:N24D:MAN5:QBM4:RD6D:APBV:OZSJ:G6LB:NUAN:MYJD:B6VS:G6HU Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

- OS (e.g. from `/etc/os-release`):
Ubuntu 22.04 LTS

- Kubernetes version: (use `kubectl version`):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-25T19:35:11Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

- Any proxies or other special environment settings?:
No

[psankar]

Closing the issue as I figured out that I had a mistake in the cluster config. Everything works fine. Sorry for the noise.