Session Stickiness doesn't work (Cookie)

[aczarnecki]

Hi,

so I tried to use haproxy for ensuring session stickiness in AWS running Docker Swarm. The call flow is as follows: AWS ELB -> Docker -> haproxy -> worker

The error I can't see any Cookie being set, and the session is not sticky at all. I'll remain in the same session, but if I leave it for half a minute I'll get another one. (I'm in the same session until tomcat close the connection or at least that what it seems like).

Additionally I can't reuse a Cookie already containing session ID like AWSALB set by AWS.

When I balance it with "source" it sticks to the same session, but at least for this solution I'm stuck with using the same server over and over again for all users.

Session stickiness is turned on in ELB, haproxy is configured like this (on manager node):

 lbab:
    image: dockercloud/haproxy
    environment:
      MODE: tcp
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 8280:8080
    networks:
      - AB
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
      placement:
        constraints: [node.role == manager]

And the worker node (tomcat):

  ab:
    image: 127.0.0.1:5000/ab
    environment:
      TCP_PORTS: 8080
      COOKIE:  SRV insert indirect preserve nocache
      SERVICE_PORTS: 8080
      VIRTUAL_HOST: "*:8080/ab*"
    #ports:
    #  - 8280:8080
    networks:
      - xCenter
      - AB
    deploy:
      mode: global
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.labels.type == ab
          - node.role == worker

docker info:

Containers: 5
 Running: 4
 Paused: 0
 Stopped: 1
Images: 4
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: oh5o6gz6fkwvqacaogb6fqsz1
 Is Manager: true
 ClusterID: w5x6m47p2xeof0rjp8iqctwal
 Managers: 2
 Nodes: 12
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.10.0.195
 Manager Addresses:
  10.10.0.195:2377
  10.10.5.150:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 4.4.36-8-default
Operating System: openSUSE Leap 42.2
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.951 GiB
Name: ip-10-10-0-195
ID: LIZB:FREV:3DEJ:6YE5:7RSY:KV4G:CBQB:QSQH:HWLK:A3HF:UE4W:ZWTM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

I'm using dockercloud/haproxy in version 1.6.6.

[aczarnecki]

Probably found the reason. This part of the code should set cookie in the route:

backend_route = ["server %s %s" % (route["container_name"], address)]
                    if is_sticky:
                        backend_route.append("cookie %s" % route["container_name"])

This is the log from HA Proxy (see that there is no cookie in the route, changed Cookie configuration to JSESSIONID prefix nocache).

INFO:haproxy:dockercloud/haproxy 1.6.6 is running outside Docker Cloud
INFO:haproxy:Haproxy is running in SwarmMode, loading HAProxy definition through docker api
INFO:haproxy:dockercloud/haproxy PID: 5
INFO:haproxy:=> Add task: Initial start - Swarm Mode
INFO:haproxy:=> Executing task: Initial start - Swarm Mode
INFO:haproxy:==========BEGIN==========
INFO:haproxy:Linked service: is_ab
INFO:haproxy:Linked container: is_ab.0.owaxrk15a6ffknchdbijku49w, is_ab.0.prqhwspig9djsfld13yuo2mab
INFO:haproxy:HAProxy configuration:
global
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  log-send-hostname
  maxconn 4096
  pidfile /var/run/haproxy.pid
  user haproxy
  group haproxy
  daemon
  stats socket /var/run/haproxy.stats level admin
  ssl-default-bind-options no-sslv3
  ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:DHE-DSS-AES128-SHA:DES-CBC3-SHA
defaults
  balance roundrobin
  log global
  mode http
  option redispatch
  option httplog
  option dontlognull
  option forwardfor
  timeout connect 5000
  timeout client 50000
  timeout server 50000
listen stats
  bind :1936
  mode http
  stats enable
  timeout connect 10s
  timeout client 1m
  timeout server 1m
  stats hide-version
  stats realm Haproxy\ Statistics
  stats uri /
  stats auth stats:stats
listen port_8080
  bind :8080
  mode tcp
  server is_ab.0.prqhwspig9djsfld13yuo2mab 10.0.4.4:8080 check inter 2000 rise 2 fall 3
  server is_ab.0.owaxrk15a6ffknchdbijku49w 10.0.4.3:8080 check inter 2000 rise 2 fall 3
backend SERVICE_is_ab
  cookie JSESSIONID prefix nocache
INFO:haproxy:Launching HAProxy
INFO:haproxy:HAProxy has been launched(PID: 11)
INFO:haproxy:===========END===========

[aczarnecki]

Ok, resolved this by switching to http (instead of tcp). Is there a way to do this in tcp?

[tifayuki]

@aczarnecki both session and cookie are only applicable to the application layer of the network mode, namely HTTP. TCP itself doesn't understand the those settings, I think.