Sticky ingress annotations don't appear to work

jpconver commented 6 years ago

NGINX Ingress controller version: image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):

NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.3 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial

Kernel (e.g. uname -a):

Linux jpconver.xcade.net 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Install tools:
Others:

What happened:

After deploying the ingress controller and default backend, I created a test ingress object based on the example, but the sticky session cookie isn't set.

What you expected to happen:

That the cookie is set

How to reproduce it (as minimally and precisely as possible):

1) created an example service using this yml (with kubectl create -f)

apiVersion: v1
kind: Service
metadata:
  name: example-service
  labels:
    app: example-service
spec:
  ports:
  - port: 80
    targetPort: 4000
  type: NodePort
  selector:
    app: example-pod
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: example-replica
  labels:
    app: example-replica
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: example-pod
    spec:
      containers:
      - name: example-container
        image: zlabjp/docker-example:release-0.1.0
        ports:
        - containerPort: 4000

2) create the ingress using this yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/affinity: cookie
    ingress.kubernetes.io/session-cookie-hash: sha1
    ingress.kubernetes.io/session-cookie-name: SERVERID
    ingress.kubernetes.io/rewrite-target: /foo
    kubernetes.io/ingress.class: "nginx"
  name: example.com
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 80
        path: /

3) Execute of the following command: curl -v http://192.168.39.121 -H 'Host: example.com', where 192.168.39.121 is the ip address of my minikube

* Rebuilt URL to: http://192.168.39.121/
*   Trying 192.168.39.121...
* Connected to 192.168.39.121 (192.168.39.121) port 80 (#0)
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.13.7
< Date: Sat, 24 Mar 2018 00:25:19 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 218
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Powered-By: Express
< ETag: W/"da-qlu/63uuaMtWFMah8GBxcA"
< 
* Connection #0 to host 192.168.39.121 left intact
Example Web ApplicationExample Web Application
Welcome to Example Web Application

4) The cookie is not set

Anything else we need to know:

I'm using minikube (version v0.25.0) and installed the ingress using: minikube addons enable ingress

contents of the nginx.conf file

daemon off;

worker_processes 4;
pid /run/nginx.pid;

worker_rlimit_nofile 261120;

worker_shutdown_timeout 10s ;

events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {

    real_ip_header      X-Forwarded-For;

    real_ip_recursive   on;

    set_real_ip_from    0.0.0.0/0;

    geoip_country       /etc/nginx/GeoIP.dat;
    geoip_city          /etc/nginx/GeoLiteCity.dat;
    geoip_proxy_recursive on;

    sendfile            on;

    aio                 threads;
    aio_write           on;

    tcp_nopush          on;
    tcp_nodelay         on;

    log_subrequest      on;

    reset_timedout_connection on;

    keepalive_timeout  75s;
    keepalive_requests 100;

    client_header_buffer_size       1k;
    client_header_timeout           60s;
    large_client_header_buffers     4 8k;
    client_body_buffer_size         8k;
    client_body_timeout             60s;

    http2_max_field_size            4k;
    http2_max_header_size           16k;

    types_hash_max_size             2048;
    server_names_hash_max_size      1024;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            128;

    proxy_headers_hash_max_size     512;
    proxy_headers_hash_bucket_size  64;

    variables_hash_bucket_size      128;
    variables_hash_max_size         2048;

    underscores_in_headers          off;
    ignore_invalid_headers          on;

    include /etc/nginx/mime.types;
    default_type text/html;

    brotli on;
    brotli_comp_level 4;
    brotli_types application/xml+rss application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;

    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
    gzip_proxied any;
    gzip_vary on;

    # Custom headers for response

    server_tokens on;

    # disable warnings
    uninitialized_variable_warn off;

    # Additional available variables:
    # $namespace
    # $ingress_name
    # $service_name
    log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

    map $request_uri $loggable {

        default 1;
    }

    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;

    error_log  /var/log/nginx/error.log notice;

    resolver 10.96.0.10 valid=30s;

    # Retain the default nginx handling of requests without a "Connection" header
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        ''               close;
    }

    map $http_x_forwarded_for $the_real_ip {

        default          $remote_addr;

    }

    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
    map $http_x_forwarded_proto $pass_access_scheme {
        default          $http_x_forwarded_proto;
        ''               $scheme;
    }

    map $http_x_forwarded_port $pass_server_port {
        default           $http_x_forwarded_port;
        ''                $server_port;
    }

    map $http_x_forwarded_host $best_http_host {
        default          $http_x_forwarded_host;
        ''               $this_host;
    }

    map $pass_server_port $pass_port {
        443              443;
        default          $pass_server_port;
    }

    # Obtain best http host
    map $http_host $this_host {
        default          $http_host;
        ''               $host;
    }

    server_name_in_redirect off;
    port_in_redirect        off;

    ssl_protocols TLSv1.2;

    # turn on session caching to drastically improve performance

    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;

    # allow configuring ssl session tickets
    ssl_session_tickets on;

    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;

    # allow configuring custom ssl ciphers
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;

    ssl_ecdh_curve auto;

    proxy_ssl_session_reuse on;

    upstream upstream-default-backend {
        # Load balance algorithm; empty for round robin, which is the default

        least_conn;

        keepalive 32;

        server 192.168.43.5:8080 max_fails=0 fail_timeout=0;

    }

    upstream default-example-service-80 {
        # Load balance algorithm; empty for round robin, which is the default

        least_conn;

        keepalive 32;

        server 192.168.43.14:4000 max_fails=0 fail_timeout=0;

    }

    upstream default-nifi-8443 {
        # Load balance algorithm; empty for round robin, which is the default

        least_conn;

        keepalive 32;

        server 192.168.43.12:8443 max_fails=0 fail_timeout=0;
        server 192.168.43.4:8443 max_fails=0 fail_timeout=0;

    }

    ## start server _
    server {
        server_name _ ;

        listen 80 default_server reuseport backlog=511;

        listen [::]:80 default_server reuseport backlog=511;

        set $proxy_upstream_name "-";

        listen 443  default_server reuseport backlog=511 ssl http2;

        listen [::]:443  default_server reuseport backlog=511 ssl http2;

        # PEM sha: 63596e90bb624d00ee88a072c3f97fb4de822f84
        ssl_certificate                         /ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /ingress-controller/ssl/default-fake-certificate.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";

        location / {

            set $proxy_upstream_name "upstream-default-backend";

            set $namespace      "";
            set $ingress_name   "";
            set $service_name   "";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            proxy_set_header ssl-client-cert        "";
            proxy_set_header ssl-client-verify      "";
            proxy_set_header ssl-client-dn          "";

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;

            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://upstream-default-backend;

        }

        # health checks in cloud providers require the use of port 80
        location /healthz {
            access_log off;
            return 200;
        }

        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }

    }
    ## end server _

    ## start server example.com
    server {
        server_name example.com ;

        listen 80;

        listen [::]:80;

        set $proxy_upstream_name "-";

        location / {

            set $proxy_upstream_name "default-example-service-80";

            set $namespace      "default";
            set $ingress_name   "example.com";
            set $service_name   "example-service";

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            proxy_set_header ssl-client-cert        "";
            proxy_set_header ssl-client-verify      "";
            proxy_set_header ssl-client-dn          "";

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;

            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass http://default-example-service-80;

        }

    }
    ## end server example.com

    ## start server nifi-main.nifi.default.svc.cluster.local
    server {
        server_name nifi-main.nifi.default.svc.cluster.local ;

        listen 80;

        listen [::]:80;

        set $proxy_upstream_name "-";

        listen 443  ssl http2;

        listen [::]:443  ssl http2;

        # PEM sha: 63596e90bb624d00ee88a072c3f97fb4de822f84
        ssl_certificate                         /ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /ingress-controller/ssl/default-fake-certificate.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";

        location / {

            set $proxy_upstream_name "default-nifi-8443";

            set $namespace      "default";
            set $ingress_name   "nifi";
            set $service_name   "nifi";

            # enforce ssl on server side
            if ($pass_access_scheme = http) {

                return 308 https://$best_http_host$request_uri;

            }

            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            proxy_set_header ssl-client-cert        "";
            proxy_set_header ssl-client-verify      "";
            proxy_set_header ssl-client-dn          "";

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;

            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";
            proxy_request_buffering                 "on";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

            proxy_pass https://default-nifi-8443;

        }

    }
    ## end server nifi-main.nifi.default.svc.cluster.local

    # default server, used for NGINX healthcheck and access to nginx stats
    server {
        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
        # Changing this value requires a change in:
        # https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
        listen 18080 default_server reuseport backlog=511;
        listen [::]:18080 default_server reuseport backlog=511;
        set $proxy_upstream_name "-";

        location /healthz {
            access_log off;
            return 200;
        }

        location /nginx_status {
            set $proxy_upstream_name "internal";

            access_log off;
            stub_status on;

        }

        location / {

            set $proxy_upstream_name "upstream-default-backend";
            proxy_pass          http://upstream-default-backend;
        }

    }
}

stream {
    log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

    access_log /var/log/nginx/access.log log_stream;

    error_log  /var/log/nginx/error.log;

    # TCP services

    # UDP services

}

oilbeater commented 6 years ago

you can update image tag to 0.12.0 to check if it works

pieterlange commented 6 years ago

You're not using the correct annotation prefix: https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.9.0

jpconver commented 6 years ago

Yes!!! You are right. The annotations that I should use need to start with: nginx. Thanks a lot!! For the record the correct way of specifying my ingress was then:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
    nginx.ingress.kubernetes.io/session-cookie-name: SERVERID
    nginx.ingress.kubernetes.io/rewrite-target: /foo
    kubernetes.io/ingress.class: "nginx"
  name: example.com
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 80
        path: /

ddydeveloper commented 5 years ago

Hi everyone!

I've come across with the same problem and desribed solution with the annotations does not work for me. I use docker desktop for windows with enabled k8s cluster, should be the same as minikube.

Can be reproduced with the following steps for the repo:

install minikube/docker for windows
install helm and init with the command: helm init
install nginx-ingress via helm: helm install stable/nginx-ingress --name my-nginx
creare a secret with the command: kubectl create secret generic mssql-secret --from-literal SA_PASSWORD="P@ssw0rd" --from-literal TASKS_DB="Server=mssql-cluster-ip-service;DataBase=Tasks;User Id=sa;Password=P@ssw0rd;Connection Timeout=30;"
use command kubectl apply -f k8s to apply deployments and ingress config

After cluster is set up try curl -i https://localhost -k command. According to the sticky session documentation there should be a header:

Set-Cookie: route=a9907b79b248140b56bb13723f72b67697baac3d; Expires=Sun, 12-Feb-17 14:11:12 GMT; Max-Age=172800; Path=/; HttpOnly

But I have no such headers in my response:

Could you advise how to solve this problem, maybe I misunderstood smt from the nginx-ingress docs, but is seems clear.

ddydeveloper commented 5 years ago

Finally I've figured out with the problem, the host should be used, an example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations: 
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
    nginx.ingress.kubernetes.io/session-cookie-name: route
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/acme-http01-edit-in-place: 'true'
spec:
  tls:
    - hosts:
        - your-host.com
        - www.your-host.com
      secretName: your-host-com
  rules:
    - host: your-host.com
      http:
        paths:
          - path: /
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api
            backend:              
              serviceName: server-cluster-ip-service
              servicePort: 5000
    - host: www.your-host.com
      http:
        paths:
          - path: /
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api
            backend:              
              serviceName: server-cluster-ip-service
              servicePort: 5000

kubernetes / ingress-nginx

Sticky ingress annotations don't appear to work #2252

Example Web Application