search

nginxinc / nginx-gateway-fabric

NGINX Gateway Fabric provides an implementation for the Gateway API using NGINX as the data plane.
Apache License 2.0
477 stars 94 forks source link

failed to call webhook: Post "https://gateway-api-admission-server.gateway-system.svc:443/validate?timeout=10s" #1284

Closed fabriziofiorucci closed 9 months ago

fabriziofiorucci commented 10 months ago

Describe the bug nginx-gateway-fabric on Kubernetes 1.28.1 fails to work unless the admission server webhook is installed. Instructions at https://github.com/nginxinc/nginx-gateway-fabric/blob/main/docs/installation.md#deploy-nginx-gateway-fabric-from-manifests state:

If you are running on Kubernetes 1.23 or 1.24, you also need to install the validating webhook. To do so, run:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml

The webhook still seems to be required on k8s 1.28.1.

To Reproduce

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
kubectl apply -f https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.0.0/crds.yaml
kubectl apply -f https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.0.0/nginx-gateway.yaml

Create the service (it's irrelevant whether ngf is published through LoadBalancer/NodePort):

# Source: nginx-gateway-fabric/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-gateway
  namespace: nginx-gateway
  labels:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
    app.kubernetes.io/version: "1.0.0"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: nginx-gateway
    app.kubernetes.io/instance: nginx-gateway
  ports: # Update the following ports to match your Gateway Listener ports
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443

Apply the "cafe" example: https://github.com/nginxinc/nginx-gateway-fabric/tree/main/examples/cafe-example

kubectl apply -f cafe.yaml
kubectl apply -f gateway.yaml

When applying gateway.yaml this is thrown:

Error from server (InternalError): error when creating "gateway.yaml": Internal error occurred: failed calling webhook "validate.gateway.networking.k8s.io": failed to call webhook: Post "https://gateway-api-admission-server.gateway-system.svc:443/validate?timeout=10s": service "gateway-api-admission-server" not found

The fix is:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/webhook-install.yaml

When the webhook is running, the Gateway and HTTPRoute manifests can be applied as expected

Expected behavior Based on https://github.com/nginxinc/nginx-gateway-fabric/blob/main/docs/installation.md#deploy-nginx-gateway-fabric-from-manifests Kubernetes 1.28.1 shouldn't need the webhook but that doesn't seem to be the case.

Your environment

$ kubectl get nodes -o wide
NAME            STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
f5-k8s-master   Ready    control-plane   81d   v1.28.1   192.168.2.46   <none>        Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.6.22
f5-k8s-node1    Ready    <none>          30d   v1.28.1   192.168.2.47   <none>        Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.6.22
f5-k8s-node2    Ready    <none>          81d   v1.28.1   192.168.2.48   <none>        Ubuntu 22.04.3 LTS   5.15.0-88-generic   containerd://1.6.22

On-prem Kubernetes running on KVM virtual machines, OS is Ubuntu 22.04.3 LTS

LoadBalancer, see the manifest above

docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/11/28 16:29:25 [notice] 21#21: using the "epoll" event method
2023/11/28 16:29:25 [notice] 21#21: nginx/1.25.2
2023/11/28 16:29:25 [notice] 21#21: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r10) 
2023/11/28 16:29:25 [notice] 21#21: OS: Linux 5.15.0-88-generic
2023/11/28 16:29:25 [notice] 21#21: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/11/28 16:29:25 [notice] 21#21: start worker processes
2023/11/28 16:29:25 [notice] 21#21: start worker process 40
2023/11/28 16:29:25 [notice] 21#21: start worker process 41
2023/11/28 16:29:25 [notice] 21#21: start worker process 42
2023/11/28 16:29:25 [notice] 21#21: start worker process 43
2023/11/28 16:29:25 [notice] 21#21: signal 1 (SIGHUP) received from 7, reconfiguring
2023/11/28 16:29:25 [notice] 21#21: reconfiguring
2023/11/28 16:29:25 [notice] 21#21: using the "epoll" event method
2023/11/28 16:29:25 [notice] 21#21: start worker processes
2023/11/28 16:29:25 [notice] 21#21: start worker process 44
2023/11/28 16:29:25 [notice] 21#21: start worker process 45
2023/11/28 16:29:25 [notice] 21#21: start worker process 46
2023/11/28 16:29:25 [notice] 21#21: start worker process 47
2023/11/28 16:29:25 [notice] 40#40: gracefully shutting down
2023/11/28 16:29:25 [notice] 41#41: gracefully shutting down
2023/11/28 16:29:25 [notice] 42#42: gracefully shutting down
2023/11/28 16:29:25 [notice] 42#42: exiting
2023/11/28 16:29:25 [notice] 43#43: gracefully shutting down
2023/11/28 16:29:25 [notice] 43#43: exiting
2023/11/28 16:29:25 [notice] 40#40: exiting
2023/11/28 16:29:25 [notice] 41#41: exiting
2023/11/28 16:29:25 [notice] 42#42: exit
2023/11/28 16:29:25 [notice] 40#40: exit

NGINX configuration after running the webhook and successfully applying the "cafe" example:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
load_module /usr/lib/nginx/modules/ngx_http_js_module.so;

worker_processes auto;

pid /var/run/nginx/nginx.pid;
error_log stderr info;

events {
  worker_connections 1024;
}

http {
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/mime.types;
  js_import /usr/lib/nginx/modules/njs/httpmatches.js;

  default_type application/octet-stream;

  proxy_headers_hash_bucket_size 512;
  proxy_headers_hash_max_size 1024;
  server_names_hash_bucket_size 256;
  server_names_hash_max_size 1024;
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  sendfile on;
  tcp_nopush on;

  server {
    listen unix:/var/run/nginx/nginx-status.sock;
    access_log off;

    location /stub_status {
        stub_status;
    }
  }
}

# configuration file /etc/nginx/conf.d/config-version.conf:

server {
    listen unix:/var/run/nginx/nginx-config-version.sock;
    access_log off;

    location /version {
        return 200 8;
    }
}

# configuration file /etc/nginx/conf.d/http.conf:

upstream default_coffee_80 {
    random two least_conn;
    zone default_coffee_80 512k;

    server 172.16.190.28:8080;
}

upstream default_tea_80 {
    random two least_conn;
    zone default_tea_80 512k;

    server 172.16.254.12:8080;
}

upstream invalid-backend-ref {
    random two least_conn;
    zone invalid-backend-ref 512k;

    server unix:/var/lib/nginx/nginx-500-server.sock;
}

server {
    listen 80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen 80;

    server_name cafe.example.com;

    location /coffee/ {

        proxy_set_header Host $gw_api_compliant_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_pass http://default_coffee_80$request_uri;
    }

    location = /coffee {

        proxy_set_header Host $gw_api_compliant_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_pass http://default_coffee_80$request_uri;
    }

    location = /tea {

        proxy_set_header Host $gw_api_compliant_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_pass http://default_tea_80$request_uri;
    }

    location / {
        return 404 "";

    }

}

server {
    listen unix:/var/lib/nginx/nginx-502-server.sock;
    access_log off;

    return 502;
}

server {
    listen unix:/var/lib/nginx/nginx-500-server.sock;
    access_log off;

    return 500;
}

# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
    '' $host;
    default $http_host;
}

# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

Additional context Add any other context about the problem here. Any log files you want to share.

sjberman commented 10 months ago

Hi @fabriziofiorucci, we have an issue open due to some mismatched versions in our docs. NGINX Gateway Fabric 1.0.0 doesn't support Gateway API v1.0.0. You need to install v0.8.1 of the Gateway API.

The note relating to k8s 1.23 and 1.24 is if you have installed v1.0.0 of the API and edge version of NGF.

sjberman commented 9 months ago

Closed by https://github.com/nginxinc/nginx-gateway-fabric/pull/1287. Docs are now available at https://docs.nginx.com/nginx-gateway-fabric/