cmush commented 2 years ago

What happened:

I've configured sticky sessions as per the documentation but my annotations are ignored and a client's subsequent http requests were randomly sent to all my pod/deployment's containers instead of the first one the client hit.

Scroll to the bottom of this issue to see nginx ingress controller logs that show the requests being routed randomly to the 5 pods.

What you expected to happen:

An affinity cookie be returned in responses, which I assume would allow...
Requests from a client be served by the same container that initially served them.
The expected behaviour is that only 1 pod's ip address would appear indicating that session affinity is active. I may be wrong in this expectation. If so, how do I achieve pod - client stickiness?

the ingress controller ignores affinity cookie annotations, fails to attach a cookie (route) and without the cookie nginx ingress is unable to keep sending client A's request to the container A that initially served them.

NGINX Ingress controller version :

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.2.0
  Build:         a2514768cd282c41f39ab06bda17efefc4bd233a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

Kubernetes version :

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8-gke.201", GitCommit:"2dca91e5224568a093c27d3589aa0a96fd3ddc9a", GitTreeState:"clean", BuildDate:"2022-05-11T18:39:02Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1

Environment:

Cloud provider or hardware configuration: GKE
OS (e.g. from /etc/os-release): Container-Optimized OS from Google

Basic cluster related info:

kubectl version (see kubernetes version above)

kubectl get nodes -o wide

gke-node-5728b-vcz2   Ready    <none>   5d23h   v1.22.8-gke.201   10.10.0.23    34.228.120.06     Container-Optimized OS from Google   5.10.90+         containerd://1.5.4
gke-node-8d933-853w   Ready    <none>   5d23h   v1.22.8-gke.201   10.10.0.24    34.120.222.214     Container-Optimized OS from Google   5.10.90+         containerd://1.5.4
gke-node-s9933-psb6   Ready    <none>   23m     v1.22.8-gke.201   10.10.0.26    35.222.06.201   Container-Optimized OS from Google   5.10.90+         containerd://1.5.4
gke-node-gr45b-75d4   Ready    <none>   5d23h   v1.22.8-gke.201   10.10.0.25    34.208.06.292     Container-Optimized OS from Google   5.10.90+         containerd://1.5.4

How was the ingress-nginx-controller installed:
- ingress nginx (1.2.1) via terraform helm provider and chart: https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx
```
resource "helm_release" "ingress_nginx" {
chart      = "ingress-nginx"
name       = "ingress-nginx"
namespace  = var.ingress_namespace
repository = "https://kubernetes.github.io/ingress-nginx"
version    = "4.1.0"
depends_on = [
kubernetes_namespace.ingress_namespace
]
```
values = [ templatefile("${path.module}/kubernetes/helm/templates/values-ingress-nginx.yaml", { controller_publishService_enabled = true controller_service_type = "LoadBalancer" }) ] }
```
- `helm ls -A | grep -i ingress-nginx`
```
ingress-nginx ingress-nginx 1 2022-06-16 09:09:12.29149 +0300 EAT deployed ingress-nginx-4.1.0 1.2.0
```
- `helm -n ingress-nginx get values ingress-nginx`
```
USER-SUPPLIED VALUES: controller.publishService.enabled: true controller.service.type: LoadBalancer
Current State of the controller:
- kubectl describe ingress-nginx
```
Name:             http-svc
Labels:           app=httpsvc
Namespace:        default
Address:          <<ip address>>
Ingress Class:    nginx
Default backend:  httpsvc-service:5000 (10.4.3.13:5000,10.4.4.11:5000,10.4.4.12:5000 + 2 more...)
TLS:
SNI routes service.io
Rules:
Host        Path  Backends
----        ----  --------
```
- - httpsvc-service:5000 (10.4.3.13:5000,10.4.4.11:5000,10.4.4.12:5000 + 2 more...) Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging nginx.ingress.kubernetes.io/affinity: cookie nginx.ingress.kubernetes.io/session-cookie-expires: 17250000 nginx.ingress.kubernetes.io/session-cookie-max-age: 17250000 nginx.ingress.kubernetes.io/session-cookie-name: route Events: Type Reason Age From Message
    
    Normal Sync 49m (x4 over 24h) nginx-ingress-controller Scheduled for sync Warning BadConfig 49m (x4 over 24h) cert-manager-ingress-shim Skipped a TLS block: spec.tls[0].secretName: Required value

Current state of ingress object:

kubectl get svc,ing -A -o wide


NAMESPACE       NAME                                         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                                     AGE     SELECTOR
cert-manager    service/cert-manager                         ClusterIP      10.7.246.234   <none>           9402/TCP                                    5d23h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
cert-manager    service/cert-manager-webhook                 ClusterIP      10.7.251.161   <none>           443/TCP                                     5d23h   app.kubernetes.io/component=webhook,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=webhook
default         service/cm-acme-http-solver-lmg8l            NodePort       10.7.252.251   <none>           500089:31608/TCP                              40h     acme.cert-manager.io/http-domain=923862623,acme.cert-manager.io/http-token=1195511342,acme.cert-manager.io/http01-solver=true
default         service/kubernetes                           ClusterIP      10.7.240.1     <none>           443/TCP                                     5d23h   <none>
default         service/httpsvc-service                     ClusterIP      10.7.248.184   <none>           5000/TCP,2525/TCP                           40h     app=httpsvc
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.7.253.234   <<ip address>>   5000:3150000/TCP,443:31461/TCP,2525:31509/TCP   5d23h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.7.243.216   <none>           443/TCP                                     5d23h   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     service/default-http-backend                 NodePort       10.7.251.181   <none>           5000:32257/TCP                                5d23h   k8s-app=glbc
kube-system     service/kube-dns                             ClusterIP      10.7.240.10    <none>           53/UDP,53/TCP                               5d23h   k8s-app=kube-dns
kube-system     service/metrics-server                       ClusterIP      10.7.252.39    <none>           443/TCP                                     5d23h   k8s-app=metrics-server

NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE default ingress.networking.k8s.io/cm-acme-http-solver-hj9mp www.service.io <> 5000 40h default ingress.networking.k8s.io/httpsvc-ingress nginx service.io <> 5000, 443 40h


- **Current state of deployment's pods**:
  - `kubectl get pods -o wide`

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES httpsvc-deployment-5689c846f4-254cr 1/1 Running 0 15h 10.4.5.20 gke-node-gr45b-75d4 httpsvc-deployment-5689c846f4-7r8hd 1/1 Running 0 62m 10.4.4.30 gke-node-8d933-853w httpsvc-deployment-5689c846f4-gtxpk 1/1 Running 0 30m 10.4.6.2 gke-node-s9933-psb6 httpsvc-deployment-5689c846f4-h6mjq 1/1 Running 0 62m 10.4.4.29 gke-node-8d933-853w httpsvc-deployment-5689c846f4-phxsm 1/1 Running 0 30m 10.4.5.21 gke-node-gr45b-75d4


**How to reproduce this issue**:
## Tested on

- docker desktop Kubernetes https://docs.docker.com/desktop/kubernetes/
- gke https://cloud.google.com/kubernetes-engine

## Install the ingress controller (v1.2.1) via helm (chart 4.1.4)

- `helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx`
- `helm install ingress-nginx ingress-nginx/ingress-nginx --version 4.1.4`

## Install an application that will act as default backend (is just an echo app)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml  
**PS: httpsvc is actually and elixir app so http-svc is just a placeholder for this issue.**

## Create an ingress

echo " apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress namespace: default annotations: cert-manager.io/cluster-issuer: "letsencrypt-staging" nginx.ingress.kubernetes.io/affinity: "cookie" nginx.ingress.kubernetes.io/affinity-mode: "persistent" nginx.ingress.kubernetes.io/session-cookie-name: "route" spec: ingressClassName: nginx defaultBackend: service: name: http-svc port: number: 5000 tls:

hosts:
- service.io secretName: elixir-phoenix-tls-secret rules:
host: service.io http: paths:
- path: / pathType: ImplementationSpecific backend: service: name: http-svc port: number: 5000
- path: /* pathType: ImplementationSpecific backend: service: name: http-svc port: number: 5000 " | kubectl apply -f -
```
## make a request
(sample curl request)
```
  ✗ curl -kIv --cookie cookie.txt --cookie-jar cookie.txt https://service.io/admin
  - Trying <>:443...
  - Connected to service.io (<>) port 443 (#0)
  - ALPN, offering h2
  - ALPN, offering http/1.1
  - successfully set certificate verify locations:
  - CAfile: /etc/ssl/cert.pem
  - CApath: none
  - (304) (OUT), TLS handshake, Client hello (1):
  - (304) (IN), TLS handshake, Server hello (2):
  - (304) (IN), TLS handshake, Unknown (8):
  - (304) (IN), TLS handshake, Certificate (11):
  - (304) (IN), TLS handshake, CERT verify (15):
  - (304) (IN), TLS handshake, Finished (20):
  - (304) (OUT), TLS handshake, Finished (20):
  - SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
  - ALPN, server accepted to use h2
  - Server certificate:
  - subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  - start date: Jun 16 06:10:30 2022 GMT
  - expire date: Jun 16 06:10:30 2023 GMT
  - issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  - SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  - Using HTTP2, server supports multiplexing
  - Connection state changed (HTTP/2 confirmed)
  - Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  - Using Stream ID: 1 (easy handle 0x14b00fc00)
    
    HEAD /admin HTTP/2 Host: service.io user-agent: curl/7.79.1 accept: /
  - Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 302 HTTP/2 302 < date: Wed, 22 Jun 2022 05:57:35 GMT date: Wed, 22 Jun 2022 05:57:35 GMT < content-type: text/html; charset=utf-8 content-type: text/html; charset=utf-8 < content-length: 79 content-length: 79 < cache-control: max-age=0, private, must-revalidate cache-control: max-age=0, private, must-revalidate < cross-origin-window-policy: deny cross-origin-window-policy: deny < location: /sessions/new location: /sessions/new < x-content-type-options: nosniff x-content-type-options: nosniff < x-download-options: noopen x-download-options: noopen < x-frame-options: SAMEORIGIN x-frame-options: SAMEORIGIN < x-permitted-cross-domain-policies: none x-permitted-cross-domain-policies: none < x-request-id: 5dbbe6bd9ecebc02557ccf1d4f65000517 x-request-id: 5dbbe6bd9ecebc02557ccf1d4f65000517 < x-xss-protection: 1; mode=block x-xss-protection: 1; mode=block
  - Added cookie _httpsvc_key="SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I" for domain service.io, path /, expire 0 < set-cookie: _httpsvc_key=SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I; path=/; HttpOnly set-cookie: _httpsvc_key=SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I; path=/; HttpOnly < strict-transport-security: max-age=1572450000; includeSubDomains strict-transport-security: max-age=1572450000; includeSubDomains

<

Connection #0 to host service.io left intact

contents of cookie.txt

✗ cat cookie.txt

Netscape HTTP Cookie File

https://curl.se/docs/http-cookies.html

This file was generated by libcurl! Edit at your own risk.

HttpOnly_service.io FALSE / FALSE 0 _httpsvc_key SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I


# Logs (With 5-pod deployment of httpsvc)

2022/06/22 05:45:26 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "POST /sessions HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:45:27 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 1051 1.018 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 92 0.001, 1.017 502, 302 7490e4174ee479341b1b937758189d66 10.10.0.24 - - [22/Jun/2022:05:45:27 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.002 [default-httpsvc-service-5000] [] 10.4.5.20:5000 79 0.003 302 8a9ab90540b7a0ea0abf7e13125000276a 10.10.0.24 - - [22/Jun/2022:05:45:27 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.004 [default-httpsvc-service-5000] [] 10.4.4.29:5000 6451 0.004 200 45f4e01b05018e1bdeb892504d4a3dff 10.10.0.24 - - [22/Jun/2022:05:45:34 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 559 0.968 [default-httpsvc-service-5000] [] 10.4.5.21:5000 92 0.968 302 852bf0bd4b35c3db0ddfa8fbcf3bfe6c 10.10.0.24 - - [22/Jun/2022:05:45:34 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 79 0.004 302 0e3b1bd2f4bf4797afd020f892d7d7dd 2022/06/22 05:45:35 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /sessions/new HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions/new", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:45:35 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 372 0.006 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 6451 0.002, 0.004 502, 200 e18320fe430b2d887cf9db0d3efb7003 10.10.0.24 - - [22/Jun/2022:05:45:39 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 558 0.946 [default-httpsvc-service-5000] [] 10.4.5.20:5000 92 0.945 302 4730cf92e3886261e1edcb0fafda645f 10.10.0.24 - - [22/Jun/2022:05:45:39 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 761 0.003 [default-httpsvc-service-5000] [] 10.4.4.29:5000 79 0.004 302 c1eb7e30cfbef2116afcb142a92512a3 10.10.0.24 - - [22/Jun/2022:05:45:39 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.002 [default-httpsvc-service-5000] [] 10.4.5.21:5000 6451 0.003 200 1037dee914ed7467940b7c850bce5ebc 10.10.0.24 - - [22/Jun/2022:05:45:44 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 609 0.942 [default-httpsvc-service-5000] [] 10.4.4.30:5000 92 0.942 302 8ca9779e3d15000012a287f219f6ab5d52 2022/06/22 05:45:44 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /admin HTTP/2.0", upstream: "http://10.4.3.16:5000/admin", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:45:44 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.004 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 79 0.002, 0.002 502, 302 4b93216bf9e8c97daca2bd566d075a7b 10.10.0.24 - - [22/Jun/2022:05:45:44 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.003 [default-httpsvc-service-5000] [] 10.4.5.20:5000 6451 0.002 200 6b8dc51073490d82671bb2f2f2f6a864 10.10.0.24 - - [22/Jun/2022:05:45:49 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 559 1.008 [default-httpsvc-service-5000] [] 10.4.4.29:5000 92 1.008 302 c19a2ce5645e650008de6fc31e7102cf5d 10.10.0.24 - - [22/Jun/2022:05:45:50 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.003 [default-httpsvc-service-5000] [] 10.4.5.21:5000 79 0.003 302 2cbbef83fbd17d54e8d97845501929f9 10.10.0.24 - - [22/Jun/2022:05:45:51 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 817 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 6451 0.004 200 d3b737936d4f1fc562d403af5fa286df 2022/06/22 05:45:55 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "POST /sessions HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:45:56 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 555 0.977 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 92 0.003, 0.973 502, 302 93ad38a7eb0ae1e82ae2ea185305ee26 10.10.0.24 - - [22/Jun/2022:05:45:56 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.002 [default-httpsvc-service-5000] [] 10.4.5.20:5000 79 0.001 302 3ba040a85000a0140a670b7c4dc8797256 10.10.0.24 - - [22/Jun/2022:05:45:56 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.002 [default-httpsvc-service-5000] [] 10.4.4.29:5000 6451 0.002 200 8d940e5ae5065444c7202f13493ca056 10.10.0.24 - - [22/Jun/2022:05:46:00 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 606 1.000 [default-httpsvc-service-5000] [] 10.4.5.21:5000 92 1.000 302 69bc14b44d8ca310d84b493e3d20c8af 10.10.0.24 - - [22/Jun/2022:05:46:01 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 79 0.003 302 fe4dac009f6f14f453cf9cbe34702e8e 2022/06/22 05:46:01 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /sessions/new HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions/new", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:46:01 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.004 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 6451 0.002, 0.002 502, 200 3ace9efeeabb77adf1f540e503ecd442 10.10.0.24 - - [22/Jun/2022:05:46:06 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 1000 0.963 [default-httpsvc-service-5000] [] 10.4.5.20:5000 92 0.963 302 f95000cd18c7376c6544621bf5363c2d21 10.10.0.24 - - [22/Jun/2022:05:46:06 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.003 [default-httpsvc-service-5000] [] 10.4.4.29:5000 79 0.004 302 c45bcc812168c6efeaf3c35dd4febf39 10.10.0.24 - - [22/Jun/2022:05:46:07 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 370 0.004 [default-httpsvc-service-5000] [] 10.4.5.21:5000 6451 0.003 200 1dab24e84dbde62ed3a3f7f1a3f74ebf 10.10.0.24 - - [22/Jun/2022:05:46:11 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 555 0.949 [default-httpsvc-service-5000] [] 10.4.4.30:5000 92 0.949 302 1b156f06ab1e3cf05a994bb687fea84f 2022/06/22 05:46:11 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /admin HTTP/2.0", upstream: "http://10.4.3.16:5000/admin", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:46:11 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 315 0.006 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 79 0.001, 0.004 502, 302 a202a2d0a11435fccabf5732675271ae 10.10.0.24 - - [22/Jun/2022:05:46:12 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 370 0.003 [default-httpsvc-service-5000] [] 10.4.5.20:5000 6451 0.002 200 50005e7b94c899d34a9c9447706aa8b684 10.10.0.24 - - [22/Jun/2022:05:46:12 +0000] "GET /favicon/apple-touch-icon-d55623182cfcc7a95f251e71ed2de47c.png?vsn=d HTTP/2.0" 499 0 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 328 0.000 [default-httpsvc-service-5000] [] 10.4.4.29:5000 0 0.000 - 9a60a60451343f5cda72b7dea0cbbc55 10.10.0.24 - - [22/Jun/2022:05:46:17 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 50009 0.964 [default-httpsvc-service-5000] [] 10.4.5.21:5000 92 0.965 302 61ec9e3e6c48bda4e9f7703b2e62725a 10.10.0.24 - - [22/Jun/2022:05:46:17 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 79 0.003 302 b92b688d67f0468fd9581f88e13870c7 2022/06/22 05:46:17 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /sessions/new HTTP/2.0", upstream: "http://10.4.3.16:5000/sessions/new", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:46:17 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.006 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 6451 0.001, 0.004 502, 200 1272477154fe5ec6bfe1181815601566 10.10.0.24 - - [22/Jun/2022:05:46:21 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 554 0.998 [default-httpsvc-service-5000] [] 10.4.5.20:5000 92 0.997 302 79679875fb10f3b3da1d26e54ccd4d7b 10.10.0.24 - - [22/Jun/2022:05:46:21 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 316 0.004 [default-httpsvc-service-5000] [] 10.4.4.29:5000 79 0.004 302 abf062d66b8b2c7b6c2500f82b092627 10.10.0.24 - - [22/Jun/2022:05:46:22 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 370 0.003 [default-httpsvc-service-5000] [] 10.4.5.21:5000 6451 0.003 200 d0d539aff365364ef7bcfa14a0fd5a85 10.10.0.24 - - [22/Jun/2022:05:46:26 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 559 0.935 [default-httpsvc-service-5000] [] 10.4.4.30:5000 92 0.935 302 1dd717f6d20c2607c0c46b94534b016a 2022/06/22 05:46:27 [error] 1203#1203: 2176182 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.24, server: service.io, request: "GET /admin HTTP/2.0", upstream: "http://10.4.3.16:5000/admin", host: "service.io", referrer: "https://service.io/sessions/new" 10.10.0.24 - - [22/Jun/2022:05:46:27 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 761 0.004 [default-httpsvc-service-5000] [] 10.4.3.16:5000, 10.4.6.2:5000 0, 79 0.002, 0.002 502, 302 666135aec577a9aab75603f11c17beb3 10.10.0.24 - - [22/Jun/2022:05:46:27 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 371 0.003 [default-httpsvc-service-5000] [] 10.4.5.20:5000 6451 0.002 200 b3582526e6fce2315188e4db4c3bcbeb 10.10.0.24 - - [22/Jun/2022:05:46:31 +0000] "POST /sessions HTTP/2.0" 302 92 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 608 0.988 [default-httpsvc-service-5000] [] 10.4.4.29:5000 92 0.988 302 ae48fc0830558250004c0ce36d9b20cbb1 10.10.0.24 - - [22/Jun/2022:05:46:31 +0000] "GET /admin HTTP/2.0" 302 79 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 317 0.003 [default-httpsvc-service-5000] [] 10.4.5.21:5000 79 0.002 302 2aaadd4ac6a7e59190d27cefaf1cffc6 10.10.0.24 - - [22/Jun/2022:05:46:31 +0000] "GET /sessions/new HTTP/2.0" 200 6451 "https://service.io/sessions/new" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0" 372 0.003 [default-httpsvc-service-5000] [] 10.4.4.30:5000 6451 0.003 200 d2a67493e1230b9fdf99686f458dae89

k8s-ci-robot commented 2 years ago

@cmush: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

longwuyuan commented 2 years ago

@cmush on slack you posted kubectl output here but I don't see it.

In any case, affinity works for me. To know why its not working for you, you can help out by showing the information that is relevant to affinity config. Things like ;

kubectl get svc,ing -A -o -wide
kubectl describe ing -A -o wide
curl command in full with -v and response output
logs of the ingress-nginx controller pod
Explain exactly how you installed Ingress-NGINX-controller i.e the complete command along with values file or other such details

On the slack message what you showed looks like 2 ingress objects. and then you showed what looked like a ingress-object with * as value for the spec hosts. You should check the output of Kubectl explain for the host field. * as value is invalid

longwuyuan commented 2 years ago

/remove-kind bug

cmush commented 2 years ago

@longwuyuan I've updated the github issue with the requested output and some more info.

longwuyuan commented 2 years ago

Not sure what problem needs to be solved, based on the information posted here.

longwuyuan commented 2 years ago

https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/

cmush commented 2 years ago

I have a deployment with 5 pods (let's name them A, B, C, D and E). Internally they are load balanced using ClusterIP service (httpsvc-service). External traffic is routed to httpsvc-service via nginx ingress controller and domain service.io.

Let's say there are 2 clients: @cmush and @longwuyuan expected behavior:

if @cmush makes a request to service.io, and pod A serves my initial request, I'd like to have pod A serve all my subsequent requests for the duration of that session.
if you @longwuyuan make a request to service.io, and pod B serves the initial request, I'd like to have pod B to serve all your subsequent requests for the duration of that session.

At the moment, our requests are randomly served by any of the pods thus affecting session management on httpsvc.

longwuyuan commented 2 years ago

I don't see a curl request's response coming from a unexpected backend

cmush commented 2 years ago

I don't understand what you mean. Is there any more information I can provide that'd make things clearer. Perhaps I can dm you and explain in detail what I'm trying to achieve?

longwuyuan commented 2 years ago

When I test, I can get response from same pod for all curl requests. So affinity works for me.

I was hoping you can write step-by-step instructions for someone to copy/paste on their minikube/kind cluster, that will demonstrate that response is not coming from same pod, for all curl requests. Your instructions have to be complete and accurate in details related to creating the pod, creating the service, creating the ingress and the curl command. The logs and curl response should show that responding pod is different for different curl requests to same url with same headers.

cmush commented 2 years ago

The steps I followed:

Install ingress nginx via helm chart: https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx (in my case I used the terrform helm provider)
Install an application that will act as backend (in my case this httpsvc which is an elixir phoenix app)
Create an ingress (with affinity annotations like the example above)
Access login route from browser, fill in login details and submit form (always fails if I'm running more than 1 pod but succeeds if just 1 pod is running)

longwuyuan commented 2 years ago

Without seeing a report from you that shows that affinity is broken, it will be effort without purpose. Can you show affinity is not working as real data and not just typed text. For example this link shows affinity working ; https://github.com/kubernetes/ingress-nginx/issues/7778#issuecomment-941438989

I have made mistakes on this earlier so trying to be base all comments on data instead of assumptions .

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

cmush commented 2 years ago

For anyone who comes looking, this was an issue with my implementation of Kustomize. It's not a bug on ingress-nginx.

adrianhelvik commented 1 year ago

@cmush Could you explain? I can't get sticky sessions to work in a new cluster with previously working config.

Rezkmike commented 1 year ago

/remove-lifecycle stale

sichiba commented 1 year ago

For anyone who comes looking, this was an issue with my implementation of Kustomize. It's not a bug on ingress-nginx.

can you please share with us what was the issue and how you fixed it because we're getting the same thing

wolf-mik commented 1 month ago

I didn't use Kustomize, still I am having same issues with session affinity using nginx, the session cookie is set correctly, but it looks like nginx for some reason doesn't respect the cookie. I used helm to install nginx and followed the documentation. All the configuration is set correctly as per documentation. I use docker-desktop as my k8s env, could that be the reason? I used minikube but still the same issue, anyone can help?

kubernetes / ingress-nginx

Sticky Sessions / Affinity Cookie not working as expected #8706

Netscape HTTP Cookie File

https://curl.se/docs/http-cookies.html

This file was generated by libcurl! Edit at your own risk.

HttpOnly_service.io FALSE / FALSE 0 _httpsvc_key SFMyNTY.g3QAAAACbQAAAA1waG9lbml4X2ZsYXNodAAAAAFtAAAABWVycm9ybQAAACRQbGVhc2Ugc2lnbiBpbiBpbiBvcmRlciB0byBjb250aW51ZS5tAAAADnVzZXJfcmV0dXJuX3RvbQAAABpodHRwczovL211Y2hlcnUudGVjaC9hZG1pbg.HLg8kJzHBxg1q8HT2CIjnPT8xKQREJOn7WQm71r-_1I