failed calling webhook "ingress-nginx-admission"

ktasper commented 1 year ago

What happened:

When trying to modify / create ingress resources I get this error, for example running this command to quickly create an ingress (Same behaviour via Helm, Kustomize, Kubectl, etc):

kubectl create ingress demo-localhost --class=nginx \
--rule="demo.localdev.me/*=demo:80"

results in this

error: failed to create ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": context deadline exceeded

What you expected to happen:

I expect an ingress resource to be created.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.6.4
  Build:         69e8833858fb6bda12a44990f1d5eaa7b13f4b75
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: v1.26.3
Kustomize Version: v4.5.7
Server Version: v1.24.10-gke.2300
WARNING: version difference between client (1.26) and server (1.24) exceeds the supported minor version skew of +/-1

Environment:

Cloud provider or hardware configuration: Google Cloud
OS (e.g. from /etc/os-release): v1.24.10-gke.2300
Kernel (e.g. uname -a): v1.24.10-gke.2300
Install tools:
- Unkown to me

Basic cluster related info:

kubectl get nodes -o wide

NAME                                                  STATUS   ROLES    AGE   VERSION            INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gke-my-cluster-non-preemptible-291c23c3-bbv9   Ready    <none>   19h   v1.24.9-gke.3200   10.0.0.20     <none>        Container-Optimized OS from Google   5.10.161+        containerd://1.6.9
gke-my-cluster-preemptible-577ffdb2-bxl4       Ready    <none>   19h   v1.24.9-gke.3200   10.0.0.88     <none>        Container-Optimized OS from Google   5.10.161+        containerd://1.6.9
gke-my-cluster-preemptible-577ffdb2-zjtn       Ready    <none>   19h   v1.24.9-gke.3200   10.0.0.87     <none>        Container-Optimized OS from Google   5.10.161+        containerd://1.6.9
gke-my-cluster-preemptible-d7654649-7228       Ready    <none>   18h   v1.24.9-gke.3200   10.0.0.89     <none>        Container-Optimized OS from Google   5.10.161+        containerd://1.6.9

How was the ingress-nginx-controller installed:

If helm was used then please show output of helm ls -A | grep -i ingress

ingress-nginx   ingress-nginx   7           2023-04-05 15:51:07.225504 +0100 BST    deployed    ingress-nginx-4.5.2     1.6.4

Current State of the controller:

kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
      app.kubernetes.io/instance=ingress-nginx
      app.kubernetes.io/managed-by=Helm
      app.kubernetes.io/name=ingress-nginx
      app.kubernetes.io/part-of=ingress-nginx
      app.kubernetes.io/version=1.6.4
      helm.sh/chart=ingress-nginx-4.5.2
Annotations:  meta.helm.sh/release-name: ingress-nginx
      meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

kubectl -n <ingresscontrollernamespace> get all -A -o wide


NAME                                            READY   STATUS      RESTARTS   AGE   IP          NODE                                              NOMINATED NODE   READINESS GATES
pod/ingress-nginx-controller-54947545cc-x6tqj   0/1     Completed   0          42h   10.1.2.4    gke-my-cluster-preemptible-577ffdb2-bxl4   <none>           <none>
pod/ingress-nginx-controller-bddb4fcdb-m4pjv    1/1     Running     0          38m   10.1.2.19   gke-my-cluster-preemptible-577ffdb2-bxl4   <none>           <none>

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/ingress-nginx-controller LoadBalancer 10.2.9.229 80:32007/TCP,443:32674/TCP 582d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.2.66.142 443/TCP 582d app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/ingress-nginx-controller 1/1 1 1 582d controller registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/ingress-nginx-controller-54947545cc 0 0 0 128d controller registry.k8s.io/ingress-nginx/controller:v1.3.1@sha256:54f7fe2c6c5a9db9a0ebf1131797109bb7a4d91f56b9b362bde2abd237dd1974 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=54947545cc replicaset.apps/ingress-nginx-controller-57cb5bf694 0 0 0 582d controller k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=57cb5bf694 replicaset.apps/ingress-nginx-controller-864689468b 0 0 0 47m controller registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=864689468b replicaset.apps/ingress-nginx-controller-bddb4fcdb 1 1 1 69m controller registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=bddb4fcdb replicaset.apps/ingress-nginx-controller-d4954dd7 0 0 0 211d controller registry.k8s.io/ingress-nginx/controller:v1.3.1@sha256:54f7fe2c6c5a9db9a0ebf1131797109bb7a4d91f56b9b362bde2abd237dd1974 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=d4954dd7

  - `kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>`

Name: ingress-nginx-controller-bddb4fcdb-m4pjv Namespace: ingress-nginx Priority: 0 Service Account: ingress-nginx Node: gke-my-cluster-preemptible-577ffdb2-bxl4/10.0.0.88 Start Time: Wed, 05 Apr 2023 15:27:20 +0100 Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx pod-template-hash=bddb4fcdb Annotations: kubectl.kubernetes.io/restartedAt: 2022-11-28T14:45:18Z Status: Running IP: 10.1.2.19 IPs: IP: 10.1.2.19 Controlled By: ReplicaSet/ingress-nginx-controller-bddb4fcdb Containers: controller: Container ID: containerd://6cfaa5436d3c89ee10799a14c685768bc539701c37dff78acb72ad2992fd12d9 Image: registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f Image ID: registry.k8s.io/ingress-nginx/controller@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: /nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller --election-id=ingress-nginx-leader --controller-class=k8s.io/ingress-nginx --ingress-class=nginx --configmap=$(POD_NAMESPACE)/ingress-nginx-controller --validating-webhook=:8443 --validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key State: Running Started: Wed, 05 Apr 2023 15:27:21 +0100 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 90Mi Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5 Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAME: ingress-nginx-controller-bddb4fcdb-m4pjv (v1:metadata.name) POD_NAMESPACE: ingress-nginx (v1:metadata.namespace) LD_PRELOAD: /usr/local/lib/libmimalloc.so Mounts: /usr/local/certificates/ from webhook-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-792rs (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: webhook-cert: Type: Secret (a volume populated by a Secret) SecretName: ingress-nginx-admission Optional: false kube-api-access-792rs: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s


- **Others**:
  - If I delete the webhook I can create ingress's that work as expected.

kubectl delete validatingwebhookconfiguration ingress-nginx-admission


  - Must be related to that, I am just not sure what to check next

k8s-ci-robot commented 1 year ago

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

longwuyuan commented 1 year ago

command line creation of ingress resource works fine. Your syntax is wrong. Please check kubectl help for the command. You can enclose the asterisk in quotes or you can exclude anything between / and =.

Please discuss in slack as there are more users and developers there.

/close

k8s-ci-robot commented 1 year ago

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9833#issuecomment-1498426966): >command line creation of ingress resource works fine. Your syntax is wrong. Please check `kubectl help` for the command. You can enclose the asterisk in quotes or you can exclude anything between / and =. > >Please discuss in slack as there are more users and developers there. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

ktasper commented 1 year ago

I have to disagree,

On a different cluster that command creates an ingress successfully. Even with known good manifests I am still getting that error.

I will try slack, thanks for looking at this.

kubectl get validatingwebhookconfigurations
NAME                                                              WEBHOOKS   AGE
cert-manager-webhook                                              1          404d
config.webhook.serving.knative.dev                                1          547d
flowcontrol-guardrails.config.common-webhooks.networking.gke.io   1          453d
gkepolicy.config.common-webhooks.networking.gke.io                1          37h
nginx-ingress-controller-ingress-nginx-admission                  1          2y175d

 kubectl create ingress demo-localhost --class=nginx \
  --rule="demo.localdev.me/*=demo:80"
ingress.networking.k8s.io/demo-localhost created

longwuyuan commented 1 year ago

In that case let's reopen. That validation normally fails it ther are packet filters.

On Thu, 6 Apr, 2023, 1:43 pm Karl Webster, @.***> wrote:

I have to disagree,

On a different cluster that command creates an ingress successfully. Even with known good manifests I am still getting that error.

I will try slack, thanks for looking at this.

kubectl get validatingwebhookconfigurations NAME WEBHOOKS AGE cert-manager-webhook 1 404dconfig.webhook.serving.knative.dev 1 547dflowcontrol-guardrails.config.common-webhooks.networking.gke.io 1 453dgkepolicy.config.common-webhooks.networking.gke.io 1 37h nginx-ingress-controller-ingress-nginx-admission 1 2y175d

kubectl create ingress demo-localhost --class=nginx \ --rule="demo.localdev.me/*=demo:80"ingress.networking.k8s.io/demo-localhost created

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/9833#issuecomment-1498663758, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWQM2HZEEPAPZVY7KX3W7Z3LLANCNFSM6AAAAAAWUGYJFQ . You are receiving this because you were mentioned.Message ID: @.***>

ktasper commented 1 year ago

Thanks,

I thought It might be a filtering issue on the firewall but I can confirm access to the service from within the cluster:

ingress-nginx-controller-bddb4fcdb-vtd5b:/etc/nginx$ curl -v https://ingress-nginx-controller-admission:443 -k
*   Trying 10.2.66.142:443...
* Connected to ingress-nginx-controller-admission (10.2.66.142) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* [CONN-0-0][CF-SSL] TLSv1.0 (OUT), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: O=nil2
*  start date: Aug 31 11:14:42 2021 GMT
*  expire date: Aug  7 11:14:42 2121 GMT
*  issuer: O=nil1
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: ingress-nginx-controller-admission
> User-Agent: curl/7.87.0
> Accept: */*
> 
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Thu, 06 Apr 2023 08:46:45 GMT
< Content-Length: 0
< 
* Connection #0 to host ingress-nginx-controller-admission left intact

ktasper commented 1 year ago

kubectl describe service ingress-nginx-controller-admission
Name:              ingress-nginx-controller-admission
Namespace:         ingress-nginx
Labels:            app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=ingress-nginx
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ingress-nginx
                   app.kubernetes.io/part-of=ingress-nginx
                   app.kubernetes.io/version=1.6.4
                   helm.sh/chart=ingress-nginx-4.5.2
Annotations:       meta.helm.sh/release-name: ingress-nginx
                   meta.helm.sh/release-namespace: ingress-nginx
Selector:          app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.2.66.142
IPs:               10.2.66.142
Port:              https-webhook  443/TCP
TargetPort:        webhook/TCP
Endpoints:         10.1.1.3:8443
Session Affinity:  None
Events:            <none>

ktasper commented 1 year ago

Issue seems to be that the local cert is invalid when trying to post to the webhook from inside the controller.

curl https://ingress-nginx-controller-admission.ingress-nginx.svc/networking/v1/ingresses
curl: (60) SSL certificate problem: unable to get local issuer certificate

ktasper commented 1 year ago

Turns out 8443 was being blocked by some rule. allowing 8443 for the ingress fixed this issue.

aayushave commented 1 year ago

@ktasper how did you fixed, can you describe more. I did understand port 8443 being block.

i am not able to resolve where to fix the rule, like in iptables or where. please reply back

ktasper commented 1 year ago

For me it was in google firewall since I was using GKE.

hebabaze commented 10 months ago

you can resolve that by opening the port : 443 and 8443 in each machine of your cluster if you are using ubuntu you can do as this : sudo ufw allow 8443 sudo ufw allow proto tcp from any to any port 8443

kubernetes / ingress-nginx

failed calling webhook "ingress-nginx-admission" #9833