search

vitabaks / postgresql_cluster

PostgreSQL High-Availability Cluster (based on "Patroni" and DCS "etcd" or "consul"). Automating with Ansible.
MIT License
1.27k stars 340 forks source link

Add patroni_maximum_lag_on_replica variable #569

Closed vitabaks closed 2 months ago

vitabaks commented 2 months ago

Issue: https://github.com/zalando/patroni/issues/1249

Introduce a new configuration variable, patroni_maximum_lag_on_replica, with a default value of "100MB". This parameter allows defining a threshold for the maximum acceptable lag for replicas. When a replica's lag surpasses this limit, it will no longer be considered for read-only traffic.

The implementation involves appending an optional ?lag=<max-lag> parameter to the health check for replica and async endpoints. By doing so, it enables excluding those replicas from load balancing whose lag exceeds the specified maximum, as determined by the patroni_maximum_lag_on_replica setting.

Documentation: Patroni Health Check Endpoints

This update ensures that only replicas capable of providing timely, consistent read-only access are considered, enhancing the reliability and accuracy of load-balanced read operations in distributed database environments managed by Patroni.

Note: If you have more strict requirements for a lag of replica, reduce the value of the 'patroni_maximum_lag_on_replica' variable or consider using synchronous replication. Conversely, if the lag doesn't matter much to your application, increase the value of the variable.

vitabaks commented 2 months ago

Test (dcs_type: "etcd", with_haproxy_load_balancing: true)

HAProxy config:

root@pgnode02:/# cat /etc/haproxy/haproxy.cfg 
global
    maxconn 100000
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    mode               tcp
    log                global
    retries            2
    timeout queue      5s
    timeout connect    5s
    timeout client     60m
    timeout server     60m
    timeout check      15s

listen stats
    mode http
    bind 10.172.0.21:7000
    stats enable
    stats uri /

listen master
    bind 10.172.0.21:5000
    maxconn 10000
    option tcplog
    option httpchk OPTIONS /primary
    http-check expect status 200
    default-server inter 3s fastinter 1s fall 3 rise 4 on-marked-down shutdown-sessions
  server pgnode01 10.172.0.20:6432 check port 8008
  server pgnode02 10.172.0.21:6432 check port 8008
  server pgnode03 10.172.0.22:6432 check port 8008

listen replicas
    bind 10.172.0.21:5001
    maxconn 10000
    option tcplog
    option httpchk OPTIONS /replica?lag=100MB
    balance roundrobin
    http-check expect status 200
    default-server inter 3s fastinter 1s fall 3 rise 2 on-marked-down shutdown-sessions
  server pgnode01 10.172.0.20:6432 check port 8008
  server pgnode02 10.172.0.21:6432 check port 8008
  server pgnode03 10.172.0.22:6432 check port 8008

listen replicas_sync
    bind 10.172.0.21:5002
    maxconn 10000
    option tcplog
    option httpchk OPTIONS /sync
    balance roundrobin
    http-check expect status 200
    default-server inter 3s fastinter 1s fall 3 rise 2 on-marked-down shutdown-sessions
  server pgnode01 10.172.0.20:6432 check port 8008
  server pgnode02 10.172.0.21:6432 check port 8008
  server pgnode03 10.172.0.22:6432 check port 8008

listen replicas_async
    bind 10.172.0.21:5003
    maxconn 10000
    option tcplog
    option httpchk OPTIONS /async?lag=100MB
    balance roundrobin
    http-check expect status 200
    default-server inter 3s fastinter 1s fall 3 rise 2 on-marked-down shutdown-sessions
  server pgnode01 10.172.0.20:6432 check port 8008
  server pgnode02 10.172.0.21:6432 check port 8008
  server pgnode03 10.172.0.22:6432 check port 8008

Test command 

# Set recovery_min_apply_delay on replica
psql -h 10.172.0.21 -p 5432 -c "alter system set recovery_min_apply_delay='2min'"
psql -h 10.172.0.21 -p 5432 -c "select pg_reload_conf()"
# Observe the replication lag
for i in {1..600}; do psql -h 10.172.0.20  -U postgres -p 5432 -c " select now(), client_addr,pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),replay_lsn)) as total_lag from pg_stat_replication"; sleep 2; done
# Run check Patroni REST API for replica
for i in {1..600}; do echo $(date); curl -I http://10.172.0.21:8008/replica?lag=100MB; sleep 2; done
# Connect to replicas (port 5001) and check listen_addresses
for i in {1..600}; do echo $(date); psql -h 10.172.0.20 -p 5001 -U postgres -c "show listen_addresses"; sleep 2; done 
# Generate data to create a lag
pgbench -h 10.172.0.20 -p 5432 -U postgres -i -s 10 postgres

Result:


             now             | client_addr | total_lag 
-----------------------------+-------------+-----------
 2024-02-03 16:36:53.5424+00 | 10.172.0.21 | 0 bytes
 2024-02-03 16:36:53.5424+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:36:55.592176+00 | 10.172.0.21 | 0 bytes
 2024-02-03 16:36:55.592176+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:36:57.640403+00 | 10.172.0.21 | 51 MB
 2024-02-03 16:36:57.640403+00 | 10.172.0.22 | 12 MB
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:36:59.700557+00 | 10.172.0.21 | 103 MB
 2024-02-03 16:36:59.700557+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:01.748605+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:01.748605+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:03.796813+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:03.796813+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:05.840725+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:05.840725+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:07.886166+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:07.886166+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:09.930055+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:09.930055+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 16:37:11.97124+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:11.97124+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:14.017889+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:14.017889+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:16.059907+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:16.059907+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 16:37:18.10554+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:18.10554+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:20.149324+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:20.149324+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:22.194485+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:22.194485+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:24.240161+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:24.240161+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:26.286059+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:26.286059+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:28.334405+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:28.334405+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:30.378289+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:30.378289+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:32.422833+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:32.422833+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:34.484061+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:34.484061+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:36.540371+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:36.540371+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:38.593894+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:38.593894+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:40.647093+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:40.647093+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 16:37:42.69976+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:42.69976+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:44.752162+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:44.752162+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:46.799584+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:46.799584+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:48.851153+00 | 10.172.0.21 | 114 MB
 2024-02-03 16:37:48.851153+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:50.916888+00 | 10.172.0.21 | 0 bytes
 2024-02-03 16:37:50.916888+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 16:37:52.976206+00 | 10.172.0.21 | 0 bytes
 2024-02-03 16:37:52.976206+00 | 10.172.0.22 | 0 bytes
(2 rows)
Sat Feb 3 16:36:43 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:45 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:47 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:49 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:51 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:53 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:55 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:57 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:36:59 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:01 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:03 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:05 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:07 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:09 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:11 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:13 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:15 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:17 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:19 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:21 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:23 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:25 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:27 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:29 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:32 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:34 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:36 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:38 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:40 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:42 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:44 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:46 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:48 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:50 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 16:37:52 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:37:54 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:37:56 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:37:58 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:38:00 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:38:02 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:38:04 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 16:38:06 UTC 2024
HTTP/1.0 200 OK
Sat Feb 3 16:36:54 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 16:36:56 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:36:58 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 16:37:01 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:03 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 16:37:05 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:07 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:09 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:11 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:13 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:15 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:17 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:19 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:21 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:23 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:25 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:27 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:29 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:31 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:33 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:35 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:37 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:39 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:41 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:43 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:45 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:47 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:50 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:52 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:54 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:37:56 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 16:37:58 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:38:00 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 16:38:02 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:38:04 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 16:38:06 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:38:08 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 16:38:10 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 16:38:12 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

We observe that during a lag period of more than 100MB (definitely in patroni_maximum_lag_on_replica), replica 10.172.0.21 is removed from read traffic balancing (port 5001) and connections are routed only to the replica (10.172.0.22) without a high replication lag. Until the lag is below the threshold, and then replica 10.172.0.21 is available again to balance the read-only traffic.

passed

vitabaks commented 2 months ago

Test (dcs_type: "consul") Consul DNS

Consul service config:

root@pgnode02:/# cat /etc/consul/conf.d/service_postgres-cluster-replica.json 
{
  "service": {
    "name": "postgres-cluster",
    "id": "postgres-cluster-replica",
    "port": 6432,
    "checks": [{"http": "http://10.172.0.21:8008/replica?lag=100MB", "interval": "2s"}, {"args": ["systemctl", "status", "pgbouncer"], "interval": "5s"}],
    "tags": ["replica"]
  }
}

Test commands:

# Set recovery_min_apply_delay on replica
psql -h 10.172.0.21 -p 5432 -U postgres -c "alter system set recovery_min_apply_delay='2min'"
psql -h 10.172.0.21 -p 5432 -U postgres -c "select pg_reload_conf()"
# Observe the replication lag
for i in {1..600}; do psql -h 10.172.0.20 -U postgres -p 5432 -c " select now(),client_addr,pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),replay_lsn)) as total_lag from pg_stat_replication"; sleep 2; done
# Run check Patroni REST API for replica
for i in {1..600}; do echo $(date); curl -I http://10.172.0.21:8008/replica?lag=100MB; sleep 2; done
# Connect to replicas and check listen_addresses
for i in {1..600}; do echo $(date); psql -h replica.postgres-cluster.service.consul -p 6432 -U postgres -c "show listen_addresses"; sleep 2; done 
# Generate data to create a lag
pgbench -h 10.172.0.20 -p 5432 -U postgres -i -s 10 postgres

Result:


              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:06.374191+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:04:06.374191+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:08.425093+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:04:08.425093+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:10.480205+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:04:10.480205+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:12.536182+00 | 10.172.0.21 | 30 MB
 2024-02-03 17:04:12.536182+00 | 10.172.0.22 | 14 MB
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:14.596481+00 | 10.172.0.21 | 103 MB
 2024-02-03 17:04:14.596481+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:16.659394+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:16.659394+00 | 10.172.0.22 | 1529 kB
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:18.709996+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:18.709996+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:20.762715+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:20.762715+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now             | client_addr | total_lag 
-----------------------------+-------------+-----------
 2024-02-03 17:04:22.8177+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:22.8177+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:24.871828+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:24.871828+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:26.933359+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:26.933359+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:28.985552+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:28.985552+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:31.037892+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:31.037892+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:33.092482+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:33.092482+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:35.142875+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:35.142875+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:37.195859+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:37.195859+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:39.248178+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:39.248178+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:41.303553+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:41.303553+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:04:43.35183+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:43.35183+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:04:45.40089+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:45.40089+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:47.450368+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:47.450368+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:49.499742+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:49.499742+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:51.559821+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:51.559821+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:53.614633+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:53.614633+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:55.660729+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:55.660729+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:57.705132+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:57.705132+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:04:59.753692+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:04:59.753692+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:01.809445+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:01.809445+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:03.859244+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:03.859244+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:05.90904+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:05.90904+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:07.958272+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:07.958272+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:10.007084+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:10.007084+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:12.057643+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:12.057643+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:14.111944+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:14.111944+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:16.173645+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:16.173645+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:18.220997+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:18.220997+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:20.270979+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:20.270979+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:22.316597+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:22.316597+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:24.366947+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:24.366947+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:26.412905+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:26.412905+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:28.468592+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:28.468592+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:30.512527+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:30.512527+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:32.56451+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:32.56451+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:34.61194+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:34.61194+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:36.667695+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:36.667695+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:38.713613+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:38.713613+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:40.770731+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:40.770731+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:42.827773+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:42.827773+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:44.892396+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:44.892396+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:46.959894+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:46.959894+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:49.012017+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:49.012017+00 | 10.172.0.22 | 0 bytes
(2 rows)

             now              | client_addr | total_lag 
------------------------------+-------------+-----------
 2024-02-03 17:05:51.07243+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:51.07243+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:53.126158+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:53.126158+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:55.181139+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:55.181139+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:57.238373+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:57.238373+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:05:59.302401+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:05:59.302401+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:01.360461+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:01.360461+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:03.413082+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:03.413082+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:05.479486+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:05.479486+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:07.545312+00 | 10.172.0.21 | 114 MB
 2024-02-03 17:06:07.545312+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:09.665645+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:06:09.665645+00 | 10.172.0.22 | 0 bytes
(2 rows)

              now              | client_addr | total_lag 
-------------------------------+-------------+-----------
 2024-02-03 17:06:11.734666+00 | 10.172.0.21 | 0 bytes
 2024-02-03 17:06:11.734666+00 | 10.172.0.22 | 0 bytes
(2 rows)

Sat Feb 3 17:04:04 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:06 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:08 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:10 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:12 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:14 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:16 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:18 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:20 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:22 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:24 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:26 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:28 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:30 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:32 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:34 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:36 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:38 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:40 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:42 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:44 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:46 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:48 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:50 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:52 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:54 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:56 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:04:58 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:00 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:02 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:04 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:06 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:08 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:10 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:12 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:14 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:16 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:18 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:20 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:22 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:24 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:26 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:28 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:30 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:32 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:34 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:36 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:38 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:40 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:42 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:44 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:46 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:49 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:51 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:53 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:55 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:57 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:05:59 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:01 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:03 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:05 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:07 UTC 2024
HTTP/1.0 503 Service Unavailable

Sat Feb 3 17:06:09 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:11 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:13 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:15 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:06:17 UTC 2024
HTTP/1.0 200 OK

Sat Feb 3 17:04:04 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:06 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:08 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:11 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:13 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:15 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:17 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:04:19 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:21 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:23 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:25 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:27 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:29 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:31 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:33 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:35 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:37 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:39 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:41 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:43 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:45 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:47 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:49 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:52 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:54 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:56 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:04:58 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:00 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:02 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:04 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:06 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:08 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:10 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:12 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:14 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:16 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:18 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:20 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:22 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:24 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:26 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:28 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:30 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:32 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:34 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:37 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:39 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:41 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:43 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:45 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:47 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:49 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:51 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:53 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:55 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:57 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:05:59 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:01 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:03 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:05 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:07 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.22,127.0.0.1
(1 row)

Sat Feb 3 17:06:09 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:11 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:13 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:16 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

Sat Feb 3 17:06:18 UTC 2024
   listen_addresses    
-----------------------
 10.172.0.21,127.0.0.1
(1 row)

We observe that during a lag period of more than 100MB (definitely in patroni_maximum_lag_on_replica), replica 10.172.0.21 is removed from read traffic balancing (port 5001) and connections are routed only to the replica (10.172.0.22) without a high replication lag. Until the lag is below the threshold, and then replica 10.172.0.21 is available again to balance the read-only traffic.

passed