nginxinc / nginx-gateway-fabric

NGINX Gateway Fabric provides an implementation for the Gateway API using NGINX as the data plane.
Apache License 2.0
501 stars 97 forks source link

The nginx-gateway's pod is not ready #1666

Closed Yelijah closed 8 months ago

Yelijah commented 8 months ago

Describe the bug After I deploy nginx gateway fabric by the helm, the nginx-gateway-fabric pod's container - nginx-gateway can't be ready.

To Reproduce Steps to reproduce the behavior:

  1. Deploy nginx gateway fabric by helm : helm install ngf oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway
  2. then the pod's status is
    $ kubectl get pod -n nginx-gateway
    NAME                                        READY   STATUS    RESTARTS   AGE
    ngf-nginx-gateway-fabric-5985484bb8-fh8dw   1/2     Running   0          21s
  3. View logs on ngf-nginx-gateway-fabric-5985484bb8-fh8dw
    $  kubectl logs ngf-nginx-gateway-fabric-5985484bb8-fh8dw -c nginx-gateway -n nginx-gateway
    {"level":"info","ts":"2024-03-12T02:21:40Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
    {"level":"info","ts":"2024-03-12T02:21:41Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
    {"level":"info","ts":"2024-03-12T02:21:42Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
    {"level":"info","ts":"2024-03-12T02:21:43Z","logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
  4. See error All logs of nginx gateway are in the attachment. nginx-gateway.log

Expected behavior My nginx-gateway pod can be ready

Your environment

worker_processes auto;

pid /var/run/nginx/nginx.pid; error_log stderr info;

events { worker_connections 1024; }

http { include /etc/nginx/conf.d/*.conf; include /etc/nginx/mime.types; js_import /usr/lib/nginx/modules/njs/httpmatches.js;

default_type application/octet-stream;

proxy_headers_hash_bucket_size 512; proxy_headers_hash_max_size 1024; server_names_hash_bucket_size 256; server_names_hash_max_size 1024; variables_hash_bucket_size 512; variables_hash_max_size 1024;

sendfile on; tcp_nopush on;

server { listen unix:/var/run/nginx/nginx-status.sock; access_log off;

location /stub_status {
    stub_status;
}

} }

configuration file /etc/nginx/conf.d/config-version.conf:

server { listen unix:/var/run/nginx/nginx-config-version.sock; access_log off;

location /version {
    return 200 1;
}

}

configuration file /etc/nginx/conf.d/http.conf:

upstream invalid-backend-ref { random two least_conn; zone invalid-backend-ref 512k;

server unix:/var/lib/nginx/nginx-500-server.sock;

}

server { listen unix:/var/lib/nginx/nginx-502-server.sock; access_log off;

return 502;

}

server { listen unix:/var/lib/nginx/nginx-500-server.sock; access_log off;

return 500;

}

Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value

of $host. We prefer $http_host because it contains the original value of the host header, which is required by the

Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use

the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.

map $http_host $gw_api_compliant_host { '' $host; default $http_host; }

Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This

allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.

map $http_upgrade $connection_upgrade { default upgrade; '' close; }

configuration file /etc/nginx/mime.types:

types { text/html html htm shtml; text/css css; text/xml xml; image/gif gif; image/jpeg jpeg jpg; application/javascript js; application/atom+xml atom; application/rss+xml rss;

text/mathml                                      mml;
text/plain                                       txt;
text/vnd.sun.j2me.app-descriptor                 jad;
text/vnd.wap.wml                                 wml;
text/x-component                                 htc;

image/avif                                       avif;
image/png                                        png;
image/svg+xml                                    svg svgz;
image/tiff                                       tif tiff;
image/vnd.wap.wbmp                               wbmp;
image/webp                                       webp;
image/x-icon                                     ico;
image/x-jng                                      jng;
image/x-ms-bmp                                   bmp;

font/woff                                        woff;
font/woff2                                       woff2;

application/java-archive                         jar war ear;
application/json                                 json;
application/mac-binhex40                         hqx;
application/msword                               doc;
application/pdf                                  pdf;
application/postscript                           ps eps ai;
application/rtf                                  rtf;
application/vnd.apple.mpegurl                    m3u8;
application/vnd.google-earth.kml+xml             kml;
application/vnd.google-earth.kmz                 kmz;
application/vnd.ms-excel                         xls;
application/vnd.ms-fontobject                    eot;
application/vnd.ms-powerpoint                    ppt;
application/vnd.oasis.opendocument.graphics      odg;
application/vnd.oasis.opendocument.presentation  odp;
application/vnd.oasis.opendocument.spreadsheet   ods;
application/vnd.oasis.opendocument.text          odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                 pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                 xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                 docx;
application/vnd.wap.wmlc                         wmlc;
application/wasm                                 wasm;
application/x-7z-compressed                      7z;
application/x-cocoa                              cco;
application/x-java-archive-diff                  jardiff;
application/x-java-jnlp-file                     jnlp;
application/x-makeself                           run;
application/x-perl                               pl pm;
application/x-pilot                              prc pdb;
application/x-rar-compressed                     rar;
application/x-redhat-package-manager             rpm;
application/x-sea                                sea;
application/x-shockwave-flash                    swf;
application/x-stuffit                            sit;
application/x-tcl                                tcl tk;
application/x-x509-ca-cert                       der pem crt;
application/x-xpinstall                          xpi;
application/xhtml+xml                            xhtml;
application/xspf+xml                             xspf;
application/zip                                  zip;

application/octet-stream                         bin exe dll;
application/octet-stream                         deb;
application/octet-stream                         dmg;
application/octet-stream                         iso img;
application/octet-stream                         msi msp msm;

audio/midi                                       mid midi kar;
audio/mpeg                                       mp3;
audio/ogg                                        ogg;
audio/x-m4a                                      m4a;
audio/x-realaudio                                ra;

video/3gpp                                       3gpp 3gp;
video/mp2t                                       ts;
video/mp4                                        mp4;
video/mpeg                                       mpeg mpg;
video/quicktime                                  mov;
video/webm                                       webm;
video/x-flv                                      flv;
video/x-m4v                                      m4v;
video/x-mng                                      mng;
video/x-ms-asf                                   asx asf;
video/x-ms-wmv                                   wmv;
video/x-msvideo                                  avi;

}



**Additional context**
Add any other context about the problem here. Any log files you want to share.
kate-osborn commented 8 months ago

Hi @Yelijah I see the following error in the nginx gateway logs:

{"level":"error","ts":"2024-03-12T02:21:19Z","logger":"eventLoop.eventHandler","msg":"Failed to update NGINX configuration","batchID":1,"error":"failed to reload NGINX: failed to send the HUP signal to NGINX main: operation not permitted"

This error would cause the problem you are seeing where the nginx-gateway Pod isn't reporting as ready.

To resolve you will need to tweak the security context of the nginx-gateway Pod. See this troubleshooting guide for details: https://docs.nginx.com/nginx-gateway-fabric/how-to/monitoring/troubleshooting/.

Yelijah commented 8 months ago

Hi @Yelijah I see the following error in the nginx gateway logs:

{"level":"error","ts":"2024-03-12T02:21:19Z","logger":"eventLoop.eventHandler","msg":"Failed to update NGINX configuration","batchID":1,"error":"failed to reload NGINX: failed to send the HUP signal to NGINX main: operation not permitted"

This error would cause the problem you are seeing where the nginx-gateway Pod isn't reporting as ready.

To resolve you will need to tweak the security context of the nginx-gateway Pod. See this troubleshooting guide for details: https://docs.nginx.com/nginx-gateway-fabric/how-to/monitoring/troubleshooting/.

Thank you for your help