Enable load balancing so multiple users can use multiple instances of the profile

[mswertz]

Motivation: if many users are using same profile then the Rock instance gets overloaed Solution: have sticky load balancing

Figuring out

• [ ] nginx
• [ ] HA proxy
• [ ] is it enough to balance on /r/session/<session_id>[?/.*]?
• [ ] what is rock server cluster?

[clemens-tolboom]

Questions

• What is "many users"?
• Sticky is session connected?
• Do we 'see' instances in the UI
• Do we start/stop in bulk?

[clemens-tolboom]

In PR #544 I made note for load balancing now moved here

Load balancing

Below sketchy docker-compose.yml file which could be added to either dev or ci from mentions PR

version: '3'
services:
  armadillo:
    image: armadillo:latest

  rock:
    image: nginx:latest
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
  rockB1:
    image: rock:latest
  rockB2:
    image: rock:latest
  rockB3:
    image: rock:latest

NGINX config

http {
    upstream backend {
        ip_hash;
        server rockB1:8080;
        server rockB2:8080;
        server rockB3:8080;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header JSESSIONID $cookie_JSESSIONID;
        }
    }
}

[StuartWheater]

I take it request forwarding will need to have Session Affinity.

[clemens-tolboom]

How to test load balancing

The sticky is not available in image: nginx:latest and learned ip_hash is not really an option (most request will probably come from same institute IP address).

Working config

Using armadillo-compose.zip from pre release v4.0.2 I can run release-test.R using changed docker-compose.yml.

Changed docker-compose.yml making rock profile a nginx proxy

```yml version: "3.4" services: armadillo: hostname: armadillo ports: - 8080:8080 build: context: '.' dockerfile: 'Dockerfile' args: JAR_FILE: '*.jar' image: molgenis/molgenis-armadillo:latest environment: LOGGING_CONFIG: 'classpath:logback-file.xml' AUDIT_LOG_PATH: '/app/logs/audit.log' SPRING_SECURITY_USER_PASSWORD: 'admin' volumes: - ${PWD}/armadillo/logs:/logs - ${PWD}/armadillo/data:/data - ${PWD}/armadillo/config:/config - /var/run/docker.sock:/var/run/docker.sock default: hostname: default image: datashield/rock-base:latest environment: DEBUG: "TRUE" xenon: hostname: xenon image: datashield/rock-dolomite-xenon:latest environment: DEBUG: "TRUE" # Armadillo think this is a rock server rock: image: nginx:latest # Only exposed for testing load balancing ports: - 8085:8085 volumes: - ./nginx.conf:/etc/nginx/nginx.conf depends_on: - rockB1 - rockB2 - rockB3 rockB1: image: datashield/rock-base:latest environment: DEBUG: "TRUE" rockB2: image: datashield/rock-base:latest rockB3: image: datashield/rock-base:latest ```

and

nginx load balancer

```conf # ./nginx.conf events {} http { upstream backend { ip_hash; server rockB1:8085; server rockB2:8085; server rockB3:8085; # sticky; } server { listen 8085; location / { proxy_pass http://backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header JSESSIONID $cookie_JSESSIONID; } } } ```

[clemens-tolboom]

HAProxy

HA Proxy may be an alternative for nginx.sticky

• http://docs.haproxy.org/2.9/configuration.html#2.7
• https://sdk.collaboraonline.com/docs/installation/Proxy_settings.html#example-with-haproxy

# ./docker-compose.yml
...
  haproxy:
    image: haproxy:2.3
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    ports:
      - 8081:80
    depends_on:
      - rockB1
      - rockB2
      - rockB3

and

HA Proxy

This is WIP as it errors on rock servers not ready so we need health check on them

simple test (round robin 2 sites

```haproxy # ./haproxy.cfg # haproxy -c -f /etc/haproxy/haproxy.cfg global log /dev/log local0 log /dev/log local1 notice # chroot /var/lib/haproxy # stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners # stats timeout 30s # user haproxy # group haproxy daemon defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 # errorfile 400 /etc/haproxy/errors/400.http # errorfile 403 /etc/haproxy/errors/403.http # errorfile 408 /etc/haproxy/errors/408.http # errorfile 500 /etc/haproxy/errors/500.http # errorfile 502 /etc/haproxy/errors/502.http # errorfile 503 /etc/haproxy/errors/503.http # errorfile 504 /etc/haproxy/errors/504.http frontend http_front bind *:80 default_backend http_back backend http_back balance roundrobin # ./haproxy.cfg # haproxy -c -f /etc/haproxy/haproxy.cfg global log /dev/log local0 log /dev/log local1 notice # chroot /var/lib/haproxy # stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners # stats timeout 30s # user haproxy # group haproxy daemon defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 # errorfile 400 /etc/haproxy/errors/400.http # errorfile 403 /etc/haproxy/errors/403.http # errorfile 408 /etc/haproxy/errors/408.http # errorfile 500 /etc/haproxy/errors/500.http # errorfile 502 /etc/haproxy/errors/502.http # errorfile 503 /etc/haproxy/errors/503.http # errorfile 504 /etc/haproxy/errors/504.http frontend http_front bind *:8085 default_backend http_back backend http_back balance roundrobin # make table of 20k entries of UUID is string of len 36 stick-table type string len 36 size 20k expire 30m # make sticky on from paths like `/r/session/{session-id}...` stick on path,regsub(/r/session/,,),regsub(/\\/.*$,,) server server1 rockB1:8085 check server server2 rockB2:8085 check server server3 rockB3:8085 check ```