nginx-ingress: occasional 503 Service Temporarily Unavailable

troian commented 7 years ago

I'm experiencing often 503 response from nginx-ingress-controller which returns as well Kubernetes Ingress Controller Fake Certificate (2) instead of provided wildcard certificate. Image is gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7

Looks like at some point nginx cannot resolve proper server_name and returns fake. But then why it ignores --default-ssl-certificate argument. Anyway I'm out of thoughts thus any help appreciated

Cluster is running at GKE

Good request


$ curl -I -v -L https://environment.trysimply.com/cluster/dashboard -k
*   Trying 104.197.245.109...
* TCP_NODELAY set
* Connected to environment.trysimply.com (104.197.245.109) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.trysimply.com
* Server certificate: Go Daddy Secure Certificate Authority - G2
* Server certificate: Go Daddy Root Certificate Authority - G2
> HEAD /cluster/dashboard HTTP/1.1
> Host: environment.trysimply.com
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
HTTP/1.1 302 Moved Temporarily
< Server: nginx/1.13.0
Server: nginx/1.13.0
< Date: Wed, 07 Jun 2017 11:12:29 GMT
Date: Wed, 07 Jun 2017 11:12:29 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 161
Content-Length: 161
< Connection: keep-alive
Connection: keep-alive
< Location: https://environment.trysimply.com/oauth2/sign_in
Location: https://environment.trysimply.com/oauth2/sign_in
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
Strict-Transport-Security: max-age=15724800; includeSubDomains;

<

Curl_http_done: called premature == 0
Connection #0 to host environment.trysimply.com left intact
Issue another request to this URL: 'https://environment.trysimply.com/oauth2/sign_in'
Found bundle for host environment.trysimply.com: 0x7ffc31c0c130 [can pipeline]
Re-using existing connection! (#0) with host environment.trysimply.com
Connected to environment.trysimply.com (104.197.245.109) port 443 (#0)

HEAD /oauth2/sign_in HTTP/1.1 Host: environment.trysimply.com User-Agent: curl/7.51.0 Accept: /

< HTTP/1.1 200 OK HTTP/1.1 200 OK < Server: nginx/1.13.0 Server: nginx/1.13.0 < Date: Wed, 07 Jun 2017 11:12:29 GMT Date: Wed, 07 Jun 2017 11:12:29 GMT < Content-Type: text/html; charset=utf-8 Content-Type: text/html; charset=utf-8 < Connection: keep-alive Connection: keep-alive < Set-Cookie: environment_oauth2_proxy=; Path=/; Domain=environment.trysimply.com; Expires=Wed, 07 Jun 2017 10:12:29 GMT; HttpOnly; Secure Set-Cookie: environment_oauth2_proxy=; Path=/; Domain=environment.trysimply.com; Expires=Wed, 07 Jun 2017 10:12:29 GMT; HttpOnly; Secure < Strict-Transport-Security: max-age=15724800; includeSubDomains; Strict-Transport-Security: max-age=15724800; includeSubDomains;

<

Curl_http_done: called premature == 0
Connection #0 to host environment.trysimply.com left intact

Bad request


$ curl -I -v -L https://environment.trysimply.com/cluster/dashboard -k
*   Trying 104.197.245.109...
* TCP_NODELAY set
* Connected to environment.trysimply.com (104.197.245.109) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: Kubernetes Ingress Controller Fake Certificate
> HEAD /cluster/dashboard HTTP/1.1
> Host: environment.trysimply.com
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 503 Service Temporarily Unavailable
HTTP/1.1 503 Service Temporarily Unavailable
< Server: nginx/1.13.0
Server: nginx/1.13.0
< Date: Wed, 07 Jun 2017 11:12:33 GMT
Date: Wed, 07 Jun 2017 11:12:33 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 213
Content-Length: 213
< Connection: keep-alive
Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
Strict-Transport-Security: max-age=15724800; includeSubDomains;

<

Curl_http_done: called premature == 0
Connection #0 to host environment.trysimply.com left intact

Configs used


apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-ingress
namespace: kube-system
data:
force-ssl-redirect: "true"
ssl-redirect: "true"
use-proxy-protocol: "false"

apiVersion: v1 kind: Service metadata: name: nginx-ingress namespace: kube-system labels: app: nginx-ingress spec: type: LoadBalancer ports:

port: 80 name: http
port: 443 name: https
port: 18443 name: vpn selector: k8s-app: nginx-ingress

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: redirected-environment.trysimply.com namespace: kube-system annotations: ingress.kubernetes.io/auth-signin: "https://environment.trysimply.com/oauth2/sign_in" ingress.kubernetes.io/auth-url: "https://environment.trysimply.com/oauth2/auth" kubernetes.io/ingress.class: "nginx" ingress.kubernetes.io/rewrite-target: / spec: tls:

hosts:
- environment.trysimply.com secretName: star-trysimply-com rules:
host: environment.trysimply.com http: paths:
- path: /cluster/dashboard backend: serviceName: kubernetes-dashboard servicePort: 80

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: oauth2-proxy namespace: kube-system annotations: kubernetes.io/ingress.class: "nginx" spec: tls:

hosts:
- environment.trysimply.com secretName: star-trysimply-com rules:
host: environment.trysimply.com http: paths:
- path: /oauth2 backend: serviceName: oauth2-proxy servicePort: 4180

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-ingress namespace: kube-system spec: replicas: 1 strategy: type: Recreate revisionHistoryLimit: 1 template: metadata: labels: k8s-app: nginx-ingress spec: terminationGracePeriodSeconds: 60 containers:

image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7 name: nginx-ingress imagePullPolicy: Always readinessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP livenessProbe: httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 timeoutSeconds: 1 env:
- name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name
- name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports:
  - containerPort: 80 hostPort: 80
  - containerPort: 443 hostPort: 443
  - containerPort: 18443 hostPort: 18443 args:
  - /nginx-ingress-controller
  - --default-backend-service=kube-system/default-http-backend
  - --tcp-services-configmap=kube-system/tcp
  - --configmap=kube-system/nginx-ingress
    - --watch-namespace=kube-system
    
    - --ingress-class=nginx
    
    - --force-namespace-isolation=true
    
    - --healthz-port=10254
    
    - --logtostderr
  - --default-ssl-certificate=kube-system/star-trysimply-com
  - --v=2

Pod nginx.conf


$ kubectl exec -it -n kube-system nginx-ingress-455914881-16zs1 -- cat /etc/nginx/nginx.conf

daemon off;

worker_processes 1; pid /run/nginx.pid;

worker_rlimit_nofile 1047552; events { multi_accept on; worker_connections 16384; use epoll; }

http { set_real_ip_from 0.0.0.0/0; real_ip_header X-Forwarded-For;

real_ip_recursive   on;

geoip_country       /etc/nginx/GeoIP.dat;
geoip_city          /etc/nginx/GeoLiteCity.dat;
geoip_proxy_recursive on;
# lua section to return proper error codes when custom pages are used
lua_package_path '.?.lua;/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/lua-resty-http/lib/?.lua;';
init_by_lua_block {
    require("error_page")
}

sendfile            on;
aio                 threads;
tcp_nopush          on;
tcp_nodelay         on;

log_subrequest      on;

reset_timedout_connection on;

keepalive_timeout  75s;
keepalive_requests 100;

client_header_buffer_size       1k;
large_client_header_buffers     4 8k;
client_body_buffer_size         8k;

http2_max_field_size            4k;
http2_max_header_size           16k;

types_hash_max_size             2048;
server_names_hash_max_size      1024;
server_names_hash_bucket_size   64;
map_hash_bucket_size            64;

underscores_in_headers          off;
ignore_invalid_headers          on;

include /etc/nginx/mime.types;
default_type text/html;
gzip on;
gzip_comp_level 5;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
gzip_proxied any;

server_tokens on;

# disable warnings
uninitialized_variable_warn off;

log_format upstreaminfo '$the_x_forwarded_for - [$the_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

map $request_uri $loggable {
    default 1;
}

access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
error_log  /var/log/nginx/error.log notice;

resolver 10.59.240.10 valid=30s;

# Retain the default nginx handling of requests without a "Connection" header
map $http_upgrade $connection_upgrade {
    default          upgrade;
    ''               close;
}

# trust http_x_forwarded_proto headers correctly indicate ssl offloading
map $http_x_forwarded_proto $pass_access_scheme {
    default          $http_x_forwarded_proto;
    ''               $scheme;
}

map $http_x_forwarded_port $pass_server_port {
   default           $http_x_forwarded_port;
   ''                $server_port;
}

map $pass_access_scheme $the_x_forwarded_for {
   default           $remote_addr;
   https             $proxy_protocol_addr;
}

map $pass_access_scheme $the_real_ip {
   default           $remote_addr;
   https             $proxy_protocol_addr;
}

# map port 442 to 443 for header X-Forwarded-Port
map $pass_server_port $pass_port {
    442              443;
    default          $pass_server_port;
}

# Map a response error watching the header Content-Type
map $http_accept $httpAccept {
    default          html;
    application/json json;
    application/xml  xml;
    text/plain       text;
}

map $httpAccept $httpReturnType {
    default          text/html;
    json             application/json;
    xml              application/xml;
    text             text/plain;
}

# Obtain best http host
map $http_host $best_http_host {
    default          $http_host;
    ''               $host;
}

server_name_in_redirect off;
port_in_redirect        off;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

# turn on session caching to drastically improve performance
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_session_timeout 10m;

# allow configuring ssl session tickets
ssl_session_tickets on;

# slightly reduce the time-to-first-byte
ssl_buffer_size 4k;

# allow configuring custom ssl ciphers
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;

ssl_ecdh_curve secp384r1;

# In case of errors try the next upstream server before returning an error
proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

proxy_ssl_session_reuse on;

upstream kube-system-kubernetes-dashboard-80 {
    # Load balance algorithm; empty for round robin, which is the default
    least_conn;
    server 10.56.3.4:9090 max_fails=0 fail_timeout=0;
}
upstream kube-system-oauth2-proxy-4180 {
    # Load balance algorithm; empty for round robin, which is the default
    least_conn;
    server 10.56.1.122:4180 max_fails=0 fail_timeout=0;
}
upstream upstream-default-backend {
    # Load balance algorithm; empty for round robin, which is the default
    least_conn;
    server 10.56.3.48:8080 max_fails=0 fail_timeout=0;
}

server {
    server_name _;
    listen 80 default_server reuseport backlog=511;
    listen [::]:80 default_server reuseport backlog=511;
    set $proxy_upstream_name "-";

    listen 442 proxy_protocol default_server reuseport backlog=511 ssl http2;
    listen [::]:442 proxy_protocol  default_server reuseport backlog=511 ssl http2;
    # PEM sha: 21cdeddc99bd37f37685a924de5fe00d1ea91465
    ssl_certificate                         /ingress-controller/ssl/kube-system-star-trysimply-com.pem;
    ssl_certificate_key                     /ingress-controller/ssl/kube-system-star-trysimply-com.pem;

    more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
    location / {
        set $proxy_upstream_name "upstream-default-backend";
        port_in_redirect off;

        client_max_body_size                    "1m";

        proxy_set_header Host                   $best_http_host;

        # Pass the extracted client certificate to the backend

        # Pass Real IP
        proxy_set_header X-Real-IP              $the_real_ip;

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;
        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Real-IP              $the_real_ip;
        proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
        proxy_set_header X-Forwarded-Host       $best_http_host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
        proxy_set_header X-Original-URI         $request_uri;
        proxy_set_header X-Scheme               $pass_access_scheme;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers

        proxy_connect_timeout                   5s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      60s;

        proxy_redirect                          off;
        proxy_buffering                         off;
        proxy_buffer_size                       "4k";
        proxy_buffers                           4 "4k";

        proxy_http_version                      1.1;

        proxy_cookie_domain                     off;
        proxy_cookie_path                       off;

        proxy_pass http://upstream-default-backend;
    }

    # health checks in cloud providers require the use of port 80
    location /healthz {
        access_log off;
        return 200;
    }

    # this is required to avoid error if nginx is being monitored
    # with an external software (like sysdig)
    location /nginx_status {
        allow 127.0.0.1;
        allow ::1;
        deny all;

        access_log off;
        stub_status on;
    }
}

server {
    server_name environment.trysimply.com;
    listen 80;
    listen [::]:80;
    set $proxy_upstream_name "-";

    listen 442 proxy_protocol ssl http2;
    listen [::]:442 proxy_protocol  ssl http2;
    # PEM sha: 21cdeddc99bd37f37685a924de5fe00d1ea91465
    ssl_certificate                         /ingress-controller/ssl/kube-system-star-trysimply-com.pem;
    ssl_certificate_key                     /ingress-controller/ssl/kube-system-star-trysimply-com.pem;

    more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
    # enforce ssl on server side
    if ($pass_access_scheme = http) {
        return 301 https://$best_http_host$request_uri;
    }
    location /oauth2 {
        set $proxy_upstream_name "kube-system-oauth2-proxy-4180";
        port_in_redirect off;

        client_max_body_size                    "1m";

        proxy_set_header Host                   $best_http_host;

        # Pass the extracted client certificate to the backend

        # Pass Real IP
        proxy_set_header X-Real-IP              $the_real_ip;

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;
        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Real-IP              $the_real_ip;
        proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
        proxy_set_header X-Forwarded-Host       $best_http_host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
        proxy_set_header X-Original-URI         $request_uri;
        proxy_set_header X-Scheme               $pass_access_scheme;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers

        proxy_connect_timeout                   5s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      60s;

        proxy_redirect                          off;
        proxy_buffering                         off;
        proxy_buffer_size                       "4k";
        proxy_buffers                           4 "4k";

        proxy_http_version                      1.1;

        proxy_cookie_domain                     off;
        proxy_cookie_path                       off;

        proxy_pass http://kube-system-oauth2-proxy-4180;
    }
    # enforce ssl on server side
    if ($pass_access_scheme = http) {
        return 301 https://$best_http_host$request_uri;
    }

    location = /_external-auth-L2NsdXN0ZXIvZGFzaGJvYXJk {
        internal;
        set $proxy_upstream_name "internal";

        proxy_pass_request_body     off;
        proxy_set_header            Content-Length "";
        proxy_pass_request_headers  on;
        proxy_set_header            Host environment.trysimply.com;
        proxy_ssl_server_name       on;

        set $target https://environment.trysimply.com/oauth2/auth;
        proxy_pass $target;
    }

    location ~* ^/cluster/dashboard\/?(?<baseuri>.*) {
        set $proxy_upstream_name "kube-system-kubernetes-dashboard-80";
        port_in_redirect off;

        # this location requires authentication
        auth_request /_external-auth-L2NsdXN0ZXIvZGFzaGJvYXJk;

        error_page 401 = https://environment.trysimply.com/oauth2/sign_in;

        client_max_body_size                    "1m";

        proxy_set_header Host                   $best_http_host;

        # Pass the extracted client certificate to the backend

        # Pass Real IP
        proxy_set_header X-Real-IP              $the_real_ip;

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;
        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Real-IP              $the_real_ip;
        proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
        proxy_set_header X-Forwarded-Host       $best_http_host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
        proxy_set_header X-Original-URI         $request_uri;
        proxy_set_header X-Scheme               $pass_access_scheme;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers

        proxy_connect_timeout                   5s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      60s;

        proxy_redirect                          off;
        proxy_buffering                         off;
        proxy_buffer_size                       "4k";
        proxy_buffers                           4 "4k";

        proxy_http_version                      1.1;

        proxy_cookie_domain                     off;
        proxy_cookie_path                       off;

rewrite /cluster/dashboard/(.*) /$1 break;
rewrite /cluster/dashboard / break;
proxy_pass http://kube-system-kubernetes-dashboard-80;

    }
    location / {
        set $proxy_upstream_name "upstream-default-backend";
        port_in_redirect off;

        client_max_body_size                    "1m";

        proxy_set_header Host                   $best_http_host;

        # Pass the extracted client certificate to the backend

        # Pass Real IP
        proxy_set_header X-Real-IP              $the_real_ip;

        # Allow websocket connections
        proxy_set_header                        Upgrade           $http_upgrade;
        proxy_set_header                        Connection        $connection_upgrade;

        proxy_set_header X-Real-IP              $the_real_ip;
        proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
        proxy_set_header X-Forwarded-Host       $best_http_host;
        proxy_set_header X-Forwarded-Port       $pass_port;
        proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
        proxy_set_header X-Original-URI         $request_uri;
        proxy_set_header X-Scheme               $pass_access_scheme;

        # mitigate HTTPoxy Vulnerability
        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
        proxy_set_header Proxy                  "";

        # Custom headers

        proxy_connect_timeout                   5s;
        proxy_send_timeout                      60s;
        proxy_read_timeout                      60s;

        proxy_redirect                          off;
        proxy_buffering                         off;
        proxy_buffer_size                       "4k";
        proxy_buffers                           4 "4k";

        proxy_http_version                      1.1;

        proxy_cookie_domain                     off;
        proxy_cookie_path                       off;

        proxy_pass http://upstream-default-backend;
    }

}
# default server, used for NGINX healthcheck and access to nginx stats
server {
    # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
    # Changing this value requires a change in:
    # https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx/command.go#L104
    listen 18080 default_server reuseport backlog=511;
    listen [::]:18080 default_server reuseport backlog=511;
    set $proxy_upstream_name "-";

    location /healthz {
        access_log off;
        return 200;
    }

    location /nginx_status {
        set $proxy_upstream_name "internal";

        access_log off;
        stub_status on;
    }

    # this location is used to extract nginx metrics
    # using prometheus.
    # TODO: enable extraction for vts module.
    location /internal_nginx_status {
        set $proxy_upstream_name "internal";

        allow 127.0.0.1;
        allow ::1;
        deny all;

        access_log off;
        stub_status on;
    }

    location / {
        set $proxy_upstream_name "upstream-default-backend";
        proxy_pass             http://upstream-default-backend;
    }

}

# default server for services without endpoints
server {
    listen 8181;
    set $proxy_upstream_name "-";

    location / {
        return 503;
    }
}

}

stream { log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

access_log /var/log/nginx/access.log log_stream;

error_log  /var/log/nginx/error.log;

# TCP services
upstream tcp-kube-system-openvpn-18443 {
    server                  10.56.1.112:443;
}

server {
    listen                  18443;
    proxy_pass              tcp-kube-system-openvpn-18443;
}

# UDP services

}

weitzj commented 7 years ago

I sometimes see the same problem and my guess is the controller does not pick up updated/new ingress rules populated in the meantime. (I think this started happening for me when going from nginx-ingress-controller:0.9.0-beta.5 to nginx-ingress-controller:0.9.0-beta.7)

The only thing working for me was to gradually restart the old nginx-ingress instances. The fresh ones work as expected.

Here is a bash-script, which does these restarts:

#!/bin/bash -
set -o nounset
BASE=$(cd "$(dirname "$0")" && pwd)
pushd "${BASE}"
for i in $(kubectl get pods -n kube-system | grep nginx-ingress-lb | awk '{print $1}')
do
  echo "will kill ${i}"
  kubectl delete "pod/${i}" -n kube-system

  echo "Waiting 30 seconds for new pod to come up before killing next old pod..."
  sleep 30
done

caseylucas commented 7 years ago

@weitzj I wonder if this may be related to https://github.com/kubernetes/ingress/issues/768 - especially if a restart fixes the problem.

aledbf commented 7 years ago

@weitzj please update the image to quay.io/aledbf/nginx-ingress-controller:0.132 (current master)

troian commented 7 years ago

@weitzj restart does not work for my case. @aledbf does your ingress 0.132 contain something specific to that issue? Anyway I'll try it soon

aledbf commented 7 years ago

@troian the fix for 768 and PRs 822, 823 and 824

weitzj commented 7 years ago

@aledbf Your image quay.io/aledbf/nginx-ingress-controller:0.132 works for me.

The steps I took:

Use your image in my_nginx_controller.yaml
kubectl apply -f my_nginx_controller.yaml
restart the nginx pods (with my bash-script from above)
Using kubectl describe pod/nginx-ingress-... to see, whether your image is in use (it is by showing git-1ea89a61

Btw.:

The nginx controller runs using the cluster-admin Role for now, since I thought RBAC might be an issue.

troian commented 7 years ago

@aledbf thanks

The issue I wonder is why it produces Fake certificate even if --default-ssl-certificate specified in argument and ingress contains only one domain with same certificate chain

weitzj commented 7 years ago

@troian I also see these 503 timeouts with the current quay.io/aledbf/nginx-ingress-controller:0.132 - but only if liveness/readiness probes did not succeed. But I guess this is the intended behaviour, which makes sense to me.

aledbf commented 7 years ago

but only if liveness/readiness probes did not succeed.

There is nothing we can do to avoid 503 in that situation

troian commented 7 years ago

@weitzj, @aledbf ok, make sense. I'm not familiar with that yet. Any particular reason they might not succeed? Even in 5 minutes after pod start One of root-cause (presumably) that chrome shows such error if ingress returns Fake Certificate

troian commented 7 years ago

Seems image quay.io/aledbf/nginx-ingress-controller:0.132 helps. Thanks everyone Resolving

DerSalvador commented 5 years ago

I sometimes see the same problem and my guess is the controller does not pick up updated/new ingress rules populated in the meantime. (I think this started happening for me when going from nginx-ingress-controller:0.9.0-beta.5 to nginx-ingress-controller:0.9.0-beta.7)

The only thing working for me was to gradually restart the old nginx-ingress instances. The fresh ones work as expected.

Here is a bash-script, which does these restarts:
#!/bin/bash -
set -o nounset
BASE=$(cd "$(dirname "$0")" && pwd)
pushd "${BASE}"
for i in $(kubectl get pods -n kube-system | grep nginx-ingress-lb | awk '{print $1}')
do
  echo "will kill ${i}"
  kubectl delete "pod/${i}" -n kube-system

  echo "Waiting 30 seconds for new pod to come up before killing next old pod..."
  sleep 30
done
works then for minikube as well with
kubectl get pods -n kube-system --selector="app.kubernetes.io/name=nginx-ingress-controller" -oname

kubernetes / ingress-nginx

nginx-ingress: occasional 503 Service Temporarily Unavailable #821

- --watch-namespace=kube-system

- --ingress-class=nginx

- --force-namespace-isolation=true

- --healthz-port=10254

- --logtostderr