search

vitabaks / postgresql_cluster

PostgreSQL High-Availability Cluster (based on "Patroni" and DCS "etcd" or "consul"). Automating with Ansible.
MIT License
1.29k stars 352 forks source link

patroni deploy problem "Check that the patroni is healthy on the replica server" #113

Closed masbolgankezderim closed 2 years ago

masbolgankezderim commented 2 years ago

Hello,

I have problem with deploying

I change only some vars: cluster_vip: "10.101.100.20"

  - { host: "10.101.100.10", port: "2379" }
  - { host: "10.101.100.11", port: "2379" }
  - { host: "10.101.100.12", port: "2379" }`

patroni_cluster_bootstrap_method: "initdb"

ok: [10.101.100.11]
ok: [10.101.100.12]
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1200 retries left).
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1200 retries left).
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1199 retries left).
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1199 retries left).

how can i resolve this problem Thank you

vitabaks commented 2 years ago

Hi @AdilkhanKTR

Please attach the Patroni log from the servers where the problem occurred.

sudo systemctl -u patroni

masbolgankezderim commented 2 years ago

Master

● patroni.service - Runners to orchestrate a high-availability PostgreSQL - patroni
   Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-08-18 09:08:29 UTC; 1h 33min ago
 Main PID: 19847 (patroni)
   CGroup: /system.slice/patroni.service
           ├─19847 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni/patroni.yml
           ├─19956 /usr/pgsql-13/bin/postgres -D /var/lib/pgsql/13/data --config-file=/var/lib/pgsql/13/data/postgresql.conf --listen_addresses=10.101.100.10,127.0.0.1 --port=5432 --cluster_name=postgr...
           ├─19958 postgres: postgres-cluster: logger 
           ├─19960 postgres: postgres-cluster: checkpointer 
           ├─19961 postgres: postgres-cluster: background writer 
           ├─19962 postgres: postgres-cluster: walwriter 
           ├─19963 postgres: postgres-cluster: autovacuum launcher 
           ├─19964 postgres: postgres-cluster: archiver last was 000000010000000000000001
           ├─19965 postgres: postgres-cluster: stats collector 
           ├─19966 postgres: postgres-cluster: logical replication launcher 
           └─19968 postgres: postgres-cluster: root postgres [local] idle

Aug 18 10:40:24 pgnode01 patroni[19847]: 2021-08-18 10:40:24,211 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:40:34 pgnode01 patroni[19847]: 2021-08-18 10:40:34,027 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:40:44 pgnode01 patroni[19847]: 2021-08-18 10:40:44,000 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:40:54 pgnode01 patroni[19847]: 2021-08-18 10:40:53,999 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:41:04 pgnode01 patroni[19847]: 2021-08-18 10:41:04,059 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:41:14 pgnode01 patroni[19847]: 2021-08-18 10:41:14,063 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:41:24 pgnode01 patroni[19847]: 2021-08-18 10:41:24,025 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:41:34 pgnode01 patroni[19847]: 2021-08-18 10:41:34,011 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:41:44 pgnode01 patroni[19847]: 2021-08-18 10:41:43,999 INFO: no action. I am (pgnode01) the leader with the lock
Aug 18 10:41:53 pgnode01 patroni[19847]: 2021-08-18 10:41:53,998 INFO: no action. I am (pgnode01) the leader with the lock

replica 1

● patroni.service - Runners to orchestrate a high-availability PostgreSQL - patroni
   Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-08-18 09:08:45 UTC; 1h 34min ago
 Main PID: 19666 (patroni)
   CGroup: /system.slice/patroni.service
           └─19666 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni/patroni.yml

Aug 18 10:43:20 pgnode02 patroni[19666]: 2021-08-18 10:43:20,786 INFO: Removing data directory: /var/lib/pgsql/13/data
Aug 18 10:43:25 pgnode02 patroni[19666]: 2021-08-18 10:43:25,720 INFO: Lock owner: pgnode01; I am pgnode02
Aug 18 10:43:25 pgnode02 patroni[19666]: 2021-08-18 10:43:25,730 INFO: trying to bootstrap from leader 'pgnode01'
Aug 18 10:43:25 pgnode02 patroni[19666]: pg_basebackup: error: FATAL:  no pg_hba.conf entry for replication connection from host "10.101.100.11", user "admin", SSL off
Aug 18 10:43:25 pgnode02 patroni[19666]: 2021-08-18 10:43:25,754 ERROR: Error when fetching backup: pg_basebackup exited with code=1
Aug 18 10:43:25 pgnode02 patroni[19666]: 2021-08-18 10:43:25,754 WARNING: Trying again in 5 seconds
Aug 18 10:43:30 pgnode02 patroni[19666]: pg_basebackup: error: FATAL:  no pg_hba.conf entry for replication connection from host "10.101.100.11", user "admin", SSL off
Aug 18 10:43:30 pgnode02 patroni[19666]: 2021-08-18 10:43:30,782 ERROR: Error when fetching backup: pg_basebackup exited with code=1
Aug 18 10:43:30 pgnode02 patroni[19666]: 2021-08-18 10:43:30,783 ERROR: failed to bootstrap from leader 'pgnode01'
Aug 18 10:43:30 pgnode02 patroni[19666]: 2021-08-18 10:43:30,783 INFO: Removing data directory: /var/lib/pgsql/13/data

replica-2

● patroni.service - Runners to orchestrate a high-availability PostgreSQL - patroni
   Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-08-18 09:08:44 UTC; 1h 36min ago
 Main PID: 19720 (patroni)
   CGroup: /system.slice/patroni.service
           └─19720 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni/patroni.yml
masbolgankezderim commented 2 years ago 
Linux pgnode02 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
vitabaks commented 2 years ago

How did you prepare the replicas? It seems to me that you used the add_pgnode.yml playbook separately, but did not prepare pg_hba.conf

Now, You can specify all nodes in inventory and execute a playbook with a tag so that all cluster nodes are added to pg_hba.conf

ansible-playbook deploy_pgcluster.yml --tag pg_hba

and then try adding replicas again.

masbolgankezderim commented 2 years ago

How did you prepare the replicas? It seems to me that you used the add_pgnode.yml playbook separately, but did not prepare pg_hba.conf

i just use ansible-playbook -i inventory.yml deploy_pgcluster

Now i need use this command with tag pg_hba?

vitabaks commented 2 years ago

for deploy_pgcluster you don't need to use this tag, it can be useful for further scaling using add_pgnode. yml

Please try to deploy the cluster again and send me the entire output of the playbook execution.

vitabaks commented 2 years ago

pg_basebackup: error: FATAL: no pg_hba.conf entry for replication connection from host "10.101.100.11", user "admin", SSL off

I noticed the user "admin", and this is a bit strange since by default we use the user "replicator" for replication. Are you sure that no third-party environment variables are set on your servers?

masbolgankezderim commented 2 years ago

for deploy_pgcluster you don't need to use this tag, it can be useful for further scaling using add_pgnode. yml

Please try to deploy the cluster again and send me the entire output of the playbook execution.


PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ****************************************************************************************************************************************

PLAY [etcd_cluster] ****************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [10.101.100.12]
ok: [10.101.100.10]
ok: [10.101.100.11]

TASK [Build a firewall_ports_dynamic_var] ******************************************************************************************************************************************************************
ok: [10.101.100.10] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=master)
ok: [10.101.100.11] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=etcd_cluster)
ok: [10.101.100.11] => (item=replica)
ok: [10.101.100.12] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=replica)

TASK [Build a firewall_rules_dynamic_var] ******************************************************************************************************************************************************************
ok: [10.101.100.10] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=master)
ok: [10.101.100.11] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=etcd_cluster)
ok: [10.101.100.11] => (item=replica)
ok: [10.101.100.12] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=replica)

TASK [ansible-role-firewall : Ensure iptables is present.] *************************************************************************************************************************************************
ok: [10.101.100.12]
ok: [10.101.100.10]
ok: [10.101.100.11]

TASK [ansible-role-firewall : Flush iptables the first time playbook runs.] ********************************************************************************************************************************
changed: [10.101.100.12]
changed: [10.101.100.10]
changed: [10.101.100.11]

TASK [ansible-role-firewall : Copy firewall script into place.] ********************************************************************************************************************************************
changed: [10.101.100.12]
changed: [10.101.100.11]
changed: [10.101.100.10]

TASK [ansible-role-firewall : Copy firewall systemd unit file into place (for systemd systems).] ***********************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [ansible-role-firewall : Load the nf_conntrack_ipv4 module] *******************************************************************************************************************************************
changed: [10.101.100.12]
changed: [10.101.100.10]
changed: [10.101.100.11]

TASK [ansible-role-firewall : sysctl | keep connections alive when enabling the firewall] ******************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.12]
changed: [10.101.100.10]

TASK [ansible-role-firewall : Configure the firewall service.] *********************************************************************************************************************************************
changed: [10.101.100.12]
changed: [10.101.100.10]
changed: [10.101.100.11]

TASK [ansible-role-firewall : Check if firewalld package is installed (on RHEL).] **************************************************************************************************************************
ok: [10.101.100.11]
ok: [10.101.100.10]
ok: [10.101.100.12]

TASK [Change hostname] *************************************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [Change hostname in /etc/hosts] ***********************************************************************************************************************************************************************
changed: [10.101.100.11] => (item=None)
changed: [10.101.100.12] => (item=None)
changed: [10.101.100.10] => (item=None)
changed: [10.101.100.12] => (item=None)
changed: [10.101.100.12]
changed: [10.101.100.10] => (item=None)
changed: [10.101.100.10]
changed: [10.101.100.11] => (item=None)
changed: [10.101.100.11]
[WARNING]: flush_handlers task does not support when conditional

RUNNING HANDLER [ansible-role-firewall : restart firewall] *************************************************************************************************************************************************
changed: [10.101.100.12]
changed: [10.101.100.11]
changed: [10.101.100.10]

TASK [Build a sysctl_conf dynamic variable] ****************************************************************************************************************************************************************
ok: [10.101.100.10] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=master)
ok: [10.101.100.11] => (item=etcd_cluster)
ok: [10.101.100.12] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=replica)
ok: [10.101.100.12] => (item=replica)

TASK [sysctl : Setting kernel parameters] ******************************************************************************************************************************************************************
changed: [10.101.100.10] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.101.100.11] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.101.100.12] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.101.100.10] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.101.100.12] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.101.100.10] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.101.100.12] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.101.100.11] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.101.100.12] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.101.100.10] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.101.100.11] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.101.100.10] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.101.100.12] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.101.100.11] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.101.100.12] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.101.100.10] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.101.100.11] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.101.100.12] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.101.100.10] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.101.100.11] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.101.100.12] => (item={'name': 'kernel.sched_migration_cost_ns', 'value': '5000000'})
changed: [10.101.100.10] => (item={'name': 'kernel.sched_migration_cost_ns', 'value': '5000000'})
changed: [10.101.100.11] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.101.100.12] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.101.100.10] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.101.100.11] => (item={'name': 'kernel.sched_migration_cost_ns', 'value': '5000000'})
changed: [10.101.100.12] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.101.100.10] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.101.100.11] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.101.100.12] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.101.100.10] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.101.100.11] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.101.100.12] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.101.100.10] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.101.100.11] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.101.100.10] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
changed: [10.101.100.12] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
changed: [10.101.100.10] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.101.100.12] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.101.100.11] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.101.100.12] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.101.100.10] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.101.100.11] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
changed: [10.101.100.10] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.101.100.12] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.101.100.11] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.101.100.10] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.101.100.12] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.101.100.11] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.101.100.11] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.101.100.11] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.12]
changed: [10.101.100.11]

TASK [Download "etcd" package] *****************************************************************************************************************************************************************************
changed: [10.101.100.10] => (item=https://github.com/etcd-io/etcd/releases/download/v3.3.25/etcd-v3.3.25-linux-amd64.tar.gz)
changed: [10.101.100.12] => (item=https://github.com/etcd-io/etcd/releases/download/v3.3.25/etcd-v3.3.25-linux-amd64.tar.gz)
changed: [10.101.100.11] => (item=https://github.com/etcd-io/etcd/releases/download/v3.3.25/etcd-v3.3.25-linux-amd64.tar.gz)

TASK [Extract "etcd" into /tmp] ****************************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] *******************************************************************************************************************************************
changed: [10.101.100.10] => (item=etcd)
changed: [10.101.100.11] => (item=etcd)
changed: [10.101.100.12] => (item=etcd)
changed: [10.101.100.10] => (item=etcdctl)
changed: [10.101.100.11] => (item=etcdctl)
changed: [10.101.100.12] => (item=etcdctl)

TASK [Add etcd user] ***************************************************************************************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.12]
changed: [10.101.100.10]

TASK [Create etcd conf directory] **************************************************************************************************************************************************************************
changed: [10.101.100.12]
changed: [10.101.100.11]
changed: [10.101.100.10]

TASK [Generate conf file "/etc/etcd/etcd.conf"] ************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.12]
changed: [10.101.100.11]

TASK [etcd : Copy systemd service file] ********************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [Enable and start etcd service] ***********************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [etcd : Wait for port 2379 to become open on the host] ************************************************************************************************************************************************
ok: [10.101.100.11]
ok: [10.101.100.12]
ok: [10.101.100.10]

TASK [Wait until the etcd cluster is healthy] **************************************************************************************************************************************************************
ok: [10.101.100.10]

TASK [etcd : cluster health] *******************************************************************************************************************************************************************************
ok: [10.101.100.10] => {
    "etcd_health_result.stdout_lines": [
        "member 32b02c5d683a304c is healthy: got healthy result from http://10.101.100.10:2379",
        "member 364cbbe6e9ac728f is healthy: got healthy result from http://10.101.100.12:2379",
        "member a23ecdbef6d60c9c is healthy: got healthy result from http://10.101.100.11:2379",
        "cluster is healthy"
    ]
}

PLAY [postgres_cluster] ************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [Build a firewall_ports_dynamic_var] ******************************************************************************************************************************************************************
ok: [10.101.100.10] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=master)
ok: [10.101.100.11] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=etcd_cluster)
ok: [10.101.100.11] => (item=replica)
ok: [10.101.100.12] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=replica)

TASK [Build a firewall_rules_dynamic_var] ******************************************************************************************************************************************************************
ok: [10.101.100.10] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=master)
ok: [10.101.100.10] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=etcd_cluster)
ok: [10.101.100.11] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=etcd_cluster)
ok: [10.101.100.11] => (item=replica)
ok: [10.101.100.12] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=replica)

TASK [ansible-role-firewall : Ensure iptables is present.] *************************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [ansible-role-firewall : Flush iptables the first time playbook runs.] ********************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [ansible-role-firewall : Copy firewall script into place.] ********************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [ansible-role-firewall : Copy firewall systemd unit file into place (for systemd systems).] ***********************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [ansible-role-firewall : Load the nf_conntrack_ipv4 module] *******************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [ansible-role-firewall : sysctl | keep connections alive when enabling the firewall] ******************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [ansible-role-firewall : Configure the firewall service.] *********************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [ansible-role-firewall : Check if firewalld package is installed (on RHEL).] **************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [Change hostname] *************************************************************************************************************************************************************************************
ok: [10.101.100.11]
ok: [10.101.100.10]
ok: [10.101.100.12]

TASK [Change hostname in /etc/hosts] ***********************************************************************************************************************************************************************
ok: [10.101.100.10] => (item=None)
ok: [10.101.100.11] => (item=None)
ok: [10.101.100.12] => (item=None)
ok: [10.101.100.10] => (item=None)
ok: [10.101.100.10]
ok: [10.101.100.11] => (item=None)
ok: [10.101.100.11]
ok: [10.101.100.12] => (item=None)
ok: [10.101.100.12]

TASK [add-repository : Remove epel-release package (if exists)] ********************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [add-repository : Get epel-release-latest rpm package] ************************************************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.12]
changed: [10.101.100.10]

TASK [add-repository : Install EPEL repository] ************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [add-repository : Install Software Collections (SCL) repository for CentOS 7] *************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.12]
changed: [10.101.100.11]

TASK [add-repository : Get pgdg-redhat-repo-latest.noarch.rpm] *********************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [add-repository : Install PostgreSQL repository] ******************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [packages : Clean yum cache] **************************************************************************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.10]
changed: [10.101.100.12]

TASK [Install system packages] *****************************************************************************************************************************************************************************
changed: [10.101.100.10] => (item=['python', 'python-devel', 'python-psycopg2', 'python-setuptools', 'libselinux-python', 'libselinux-python3', 'libsemanage-python', 'policycoreutils-python'])
ok: [10.101.100.10] => (item=python3)
changed: [10.101.100.11] => (item=['python', 'python-devel', 'python-psycopg2', 'python-setuptools', 'libselinux-python', 'libselinux-python3', 'libsemanage-python', 'policycoreutils-python'])
ok: [10.101.100.11] => (item=python3)
changed: [10.101.100.10] => (item=python3-devel)
changed: [10.101.100.12] => (item=['python', 'python-devel', 'python-psycopg2', 'python-setuptools', 'libselinux-python', 'libselinux-python3', 'libsemanage-python', 'policycoreutils-python'])
ok: [10.101.100.12] => (item=python3)
changed: [10.101.100.10] => (item=python3-psycopg2)
ok: [10.101.100.10] => (item=python3-setuptools)
ok: [10.101.100.10] => (item=python3-pip)
changed: [10.101.100.10] => (item=curl)
changed: [10.101.100.11] => (item=python3-devel)
ok: [10.101.100.10] => (item=less)
changed: [10.101.100.11] => (item=python3-psycopg2)
changed: [10.101.100.10] => (item=sudo)
ok: [10.101.100.11] => (item=python3-setuptools)
changed: [10.101.100.12] => (item=python3-devel)
ok: [10.101.100.11] => (item=python3-pip)
changed: [10.101.100.10] => (item=vim)
changed: [10.101.100.12] => (item=python3-psycopg2)
changed: [10.101.100.11] => (item=curl)
ok: [10.101.100.12] => (item=python3-setuptools)
ok: [10.101.100.11] => (item=less)
ok: [10.101.100.12] => (item=python3-pip)
changed: [10.101.100.11] => (item=sudo)
changed: [10.101.100.10] => (item=gcc)
changed: [10.101.100.12] => (item=curl)
ok: [10.101.100.12] => (item=less)
changed: [10.101.100.11] => (item=vim)
changed: [10.101.100.10] => (item=jq)
ok: [10.101.100.10] => (item=iptables)
ok: [10.101.100.10] => (item=acl)
changed: [10.101.100.12] => (item=sudo)
changed: [10.101.100.12] => (item=vim)
changed: [10.101.100.11] => (item=gcc)
changed: [10.101.100.11] => (item=jq)
ok: [10.101.100.11] => (item=iptables)
ok: [10.101.100.11] => (item=acl)
changed: [10.101.100.12] => (item=gcc)
changed: [10.101.100.12] => (item=jq)
ok: [10.101.100.12] => (item=iptables)
ok: [10.101.100.12] => (item=acl)

TASK [Install PostgreSQL packages] *************************************************************************************************************************************************************************
changed: [10.101.100.10] => (item=postgresql13)
changed: [10.101.100.11] => (item=postgresql13)
changed: [10.101.100.10] => (item=postgresql13-server)
changed: [10.101.100.12] => (item=postgresql13)
changed: [10.101.100.11] => (item=postgresql13-server)
changed: [10.101.100.10] => (item=postgresql13-contrib)
changed: [10.101.100.12] => (item=postgresql13-server)
changed: [10.101.100.11] => (item=postgresql13-contrib)
changed: [10.101.100.12] => (item=postgresql13-contrib)
changed: [10.101.100.10] => (item=postgresql13-devel)
changed: [10.101.100.12] => (item=postgresql13-devel)
changed: [10.101.100.11] => (item=postgresql13-devel)

TASK [Add user to /etc/sudoers.d/] *************************************************************************************************************************************************************************
changed: [10.101.100.12] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.101.100.11] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.101.100.10] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [Build a sysctl_conf dynamic variable] ****************************************************************************************************************************************************************
ok: [10.101.100.10] => (item=etcd_cluster)
ok: [10.101.100.10] => (item=master)
ok: [10.101.100.10] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=etcd_cluster)
ok: [10.101.100.12] => (item=etcd_cluster)
ok: [10.101.100.11] => (item=postgres_cluster)
ok: [10.101.100.12] => (item=postgres_cluster)
ok: [10.101.100.11] => (item=replica)
ok: [10.101.100.12] => (item=replica)

TASK [sysctl : Setting kernel parameters] ******************************************************************************************************************************************************************
ok: [10.101.100.10] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.101.100.11] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.101.100.12] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.101.100.10] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.101.100.11] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.101.100.12] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.101.100.10] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.101.100.11] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.101.100.12] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.101.100.10] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.101.100.11] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.101.100.12] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.101.100.10] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.101.100.11] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.101.100.12] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.101.100.10] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.101.100.11] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.101.100.12] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.101.100.10] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.101.100.11] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.101.100.12] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.101.100.10] => (item={'name': 'kernel.sched_migration_cost_ns', 'value': '5000000'})
ok: [10.101.100.11] => (item={'name': 'kernel.sched_migration_cost_ns', 'value': '5000000'})
ok: [10.101.100.12] => (item={'name': 'kernel.sched_migration_cost_ns', 'value': '5000000'})
ok: [10.101.100.10] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.101.100.11] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.101.100.12] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.101.100.10] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.101.100.11] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.101.100.12] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.101.100.10] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.101.100.11] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.101.100.12] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.101.100.10] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.101.100.11] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.101.100.12] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.101.100.10] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
ok: [10.101.100.11] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
ok: [10.101.100.12] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
ok: [10.101.100.10] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.101.100.11] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.101.100.12] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.101.100.10] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.101.100.11] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.101.100.12] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.101.100.10] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.101.100.11] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.101.100.12] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.101.100.10] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.101.100.11] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.101.100.12] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ********************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.12]
changed: [10.101.100.10]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *****************************************************************************************************************************************
changed: [10.101.100.11] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.101.100.12] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.101.100.10] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.101.100.12] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.101.100.11] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.101.100.10] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [Generate locales] ************************************************************************************************************************************************************************************
ok: [10.101.100.11] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.101.100.12] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.101.100.10] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.101.100.11] => (item={'language_country': 'ru_RU', 'encoding': 'UTF-8'})
ok: [10.101.100.10] => (item={'language_country': 'ru_RU', 'encoding': 'UTF-8'})
ok: [10.101.100.12] => (item={'language_country': 'ru_RU', 'encoding': 'UTF-8'})

TASK [locales : Check that the /etc/locale.conf exists] ****************************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [locales : Set locale "en_US.utf-8" into /etc/locale.conf] ********************************************************************************************************************************************
changed: [10.101.100.11] => (item=LANG=en_US.utf-8)
changed: [10.101.100.10] => (item=LANG=en_US.utf-8)
changed: [10.101.100.12] => (item=LANG=en_US.utf-8)
changed: [10.101.100.11] => (item=LC_ALL=en_US.utf-8)
changed: [10.101.100.12] => (item=LC_ALL=en_US.utf-8)
changed: [10.101.100.10] => (item=LC_ALL=en_US.utf-8)

TASK [locales : Set locale "en_US.utf-8" into /etc/environment] ********************************************************************************************************************************************
changed: [10.101.100.10] => (item=export LANG=en_US.utf-8)
changed: [10.101.100.11] => (item=export LANG=en_US.utf-8)
changed: [10.101.100.12] => (item=export LANG=en_US.utf-8)
changed: [10.101.100.10] => (item=export LC_ALL=en_US.utf-8)
changed: [10.101.100.11] => (item=export LC_ALL=en_US.utf-8)
changed: [10.101.100.12] => (item=export LC_ALL=en_US.utf-8)

PLAY [balancers] *******************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.22]
ok: [10.101.100.23]

TASK [Build a firewall_ports_dynamic_var] ******************************************************************************************************************************************************************
ok: [10.101.100.21] => (item=balancers)
ok: [10.101.100.22] => (item=balancers)
ok: [10.101.100.23] => (item=balancers)

TASK [Build a firewall_rules_dynamic_var] ******************************************************************************************************************************************************************
ok: [10.101.100.21] => (item=balancers)
ok: [10.101.100.22] => (item=balancers)
ok: [10.101.100.23] => (item=balancers)

TASK [ansible-role-firewall : Ensure iptables is present.] *************************************************************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.23]
ok: [10.101.100.22]

TASK [ansible-role-firewall : Flush iptables the first time playbook runs.] ********************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [ansible-role-firewall : Copy firewall script into place.] ********************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [ansible-role-firewall : Copy firewall systemd unit file into place (for systemd systems).] ***********************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [ansible-role-firewall : Load the nf_conntrack_ipv4 module] *******************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [ansible-role-firewall : sysctl | keep connections alive when enabling the firewall] ******************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [ansible-role-firewall : Configure the firewall service.] *********************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [ansible-role-firewall : Check if firewalld package is installed (on RHEL).] **************************************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.23]
ok: [10.101.100.22]

RUNNING HANDLER [ansible-role-firewall : restart firewall] *************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [Build a sysctl_conf dynamic variable] ****************************************************************************************************************************************************************
ok: [10.101.100.21] => (item=balancers)
ok: [10.101.100.22] => (item=balancers)
ok: [10.101.100.23] => (item=balancers)

TASK [sysctl : Setting kernel parameters] ******************************************************************************************************************************************************************
changed: [10.101.100.21] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.101.100.22] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.101.100.23] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.101.100.21] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.101.100.23] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.101.100.22] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.101.100.21] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.101.100.23] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.101.100.22] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.101.100.23] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
changed: [10.101.100.22] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
changed: [10.101.100.23] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.101.100.22] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.101.100.21] => (item={'name': 'net.netfilter.nf_conntrack_max', 'value': '1048576'})
changed: [10.101.100.23] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.101.100.21] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.101.100.22] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.101.100.23] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.101.100.21] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.101.100.22] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.101.100.21] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.101.100.23] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.101.100.22] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.101.100.21] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [haproxy : Install Software Collections (SCL) repository for CentOS 7] ********************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [Install HAProxy v1.8 (rh-haproxy18 package)] *********************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [create a symlink "/usr/sbin/haproxy"] ****************************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [haproxy : Make sure the kernel parameter "net.ipv4.ip_nonlocal_bind" are enabled] ********************************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.22]
ok: [10.101.100.23]

TASK [Add haproxy group] ***********************************************************************************************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.22]
ok: [10.101.100.23]

TASK [Add haproxy user] ************************************************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [haproxy : Create directories] ************************************************************************************************************************************************************************
changed: [10.101.100.21] => (item=/etc/haproxy)
changed: [10.101.100.22] => (item=/etc/haproxy)
changed: [10.101.100.23] => (item=/etc/haproxy)
changed: [10.101.100.21] => (item=/var/run/haproxy)
changed: [10.101.100.22] => (item=/var/run/haproxy)
changed: [10.101.100.23] => (item=/var/run/haproxy)
changed: [10.101.100.21] => (item=/var/lib/haproxy/dev)
changed: [10.101.100.22] => (item=/var/lib/haproxy/dev)
changed: [10.101.100.23] => (item=/var/lib/haproxy/dev)

TASK [Generate conf file "/etc/haproxy/haproxy.cfg"] *******************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [Generate systemd service file "/etc/systemd/system/haproxy.service"] *********************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [haproxy : selinux | make sure the libsemanage-python, policycoreutils-python packages is present] ****************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.22]
ok: [10.101.100.23]

TASK [selinux | set haproxy_connect_any flag to enable tcp connections] ************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [selinux | change the haproxy_t domain to permissive] *************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [Download and copy "confd" binary file to /usr/local/bin/] ********************************************************************************************************************************************
changed: [10.101.100.23] => (item=https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64)
changed: [10.101.100.22] => (item=https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64)
changed: [10.101.100.21] => (item=https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64)

TASK [confd : Create conf directories] *********************************************************************************************************************************************************************
changed: [10.101.100.21] => (item=/etc/confd/conf.d)
changed: [10.101.100.22] => (item=/etc/confd/conf.d)
changed: [10.101.100.23] => (item=/etc/confd/conf.d)
changed: [10.101.100.21] => (item=/etc/confd/templates)
changed: [10.101.100.22] => (item=/etc/confd/templates)
changed: [10.101.100.23] => (item=/etc/confd/templates)

TASK [Generate conf file "/etc/confd/confd.toml"] **********************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [Generate conf file "/etc/confd/conf.d/haproxy.toml"] *************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [Generate template "/etc/confd/templates/haproxy.tmpl"] ***********************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [confd : Copy systemd service file] *******************************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [Install keepalived packages] *************************************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [keepalived : Make sure the kernel parameters "net.ipv4.ip_nonlocal_bind", "net.ipv4.ip_forward" are enabled] *****************************************************************************************
ok: [10.101.100.21] => (item=net.ipv4.ip_nonlocal_bind)
ok: [10.101.100.22] => (item=net.ipv4.ip_nonlocal_bind)
ok: [10.101.100.23] => (item=net.ipv4.ip_nonlocal_bind)
ok: [10.101.100.21] => (item=net.ipv4.ip_forward)
ok: [10.101.100.22] => (item=net.ipv4.ip_forward)
ok: [10.101.100.23] => (item=net.ipv4.ip_forward)

TASK [Make sure the "/usr/libexec/keepalived" directory exists] ********************************************************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.22]
ok: [10.101.100.23]

TASK [Create vrrp_script "/usr/libexec/keepalived/haproxy_check.sh"] ***************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

TASK [Generate conf file "/etc/keepalived/keepalived.conf"] ************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

TASK [selinux | change the keepalived_t domain to permissive] **********************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

RUNNING HANDLER [Restart haproxy service] ******************************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

RUNNING HANDLER [haproxy : Check HAProxy is started and accepting connections] *****************************************************************************************************************************
ok: [10.101.100.21]
ok: [10.101.100.23]
ok: [10.101.100.22]

RUNNING HANDLER [Restart confd service] ********************************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.23]
changed: [10.101.100.22]

RUNNING HANDLER [Restart keepalived service] ***************************************************************************************************************************************************************
changed: [10.101.100.21]
changed: [10.101.100.22]
changed: [10.101.100.23]

RUNNING HANDLER [keepalived : Wait for the cluster ip address (VIP) "10.101.100.20" is running] ************************************************************************************************************
ok: [10.101.100.23]
ok: [10.101.100.22]
ok: [10.101.100.21]

PLAY [pgbackrest:postgres_cluster] *************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

PLAY [postgres_cluster] ************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [Copy patroni requirements.txt file] ******************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [patroni : Install setuptools] ************************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [patroni : Install requirements] **********************************************************************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.10]
changed: [10.101.100.12]

TASK [Install patroni] *************************************************************************************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.10]
changed: [10.101.100.12]

TASK [patroni : Create conf directory] *********************************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [Generate conf file "/etc/patroni/patroni.yml"] *******************************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [Copy systemd service file "/etc/systemd/system/patroni.service"] *************************************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [patroni : Prepare PostgreSQL | create statistics directory (if not already exists)] ******************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [patroni : Prepare PostgreSQL | mount the statistics directory in memory (tmpfs)] *********************************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] **************************************************************************************************
changed: [10.101.100.10]
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/pgsql/13/data" exists] **************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [patroni : Prepare PostgreSQL | check that data directory "/var/lib/pgsql/13/data" is not initialized] ************************************************************************************************
ok: [10.101.100.10]
ok: [10.101.100.11]
ok: [10.101.100.12]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/pgsql/13/data" is empty] *******************************************************************************************************
changed: [10.101.100.10] => (item=absent)
changed: [10.101.100.11] => (item=absent)
changed: [10.101.100.12] => (item=absent)
changed: [10.101.100.10] => (item=directory)
changed: [10.101.100.11] => (item=directory)
changed: [10.101.100.12] => (item=directory)

TASK [Start patroni service on the Master server] **********************************************************************************************************************************************************
changed: [10.101.100.10]

TASK [patroni : Wait for port 8008 to become open on the host] *********************************************************************************************************************************************
ok: [10.101.100.10]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] ***************************************************************************************************************************
[WARNING]: Module remote_tmp /tmp/postgres/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the
correct permissions manually
ok: [10.101.100.10]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] *******************************************************************************************************************
ok: [10.101.100.10]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *************************************************************************************************************************************************
changed: [10.101.100.10]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *************************************************************************************************************************************
ok: [10.101.100.10]

TASK [Start patroni service on Replica servers] ************************************************************************************************************************************************************
changed: [10.101.100.11]
changed: [10.101.100.12]

TASK [patroni : Wait for port 8008 to become open on the host] *********************************************************************************************************************************************
ok: [10.101.100.11]
ok: [10.101.100.12]
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1200 retries left).
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1200 retries left).
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1199 retries left).
FAILED - RETRYING: Check that the patroni is healthy on the replica server (1199 retries left).
masbolgankezderim commented 2 years ago

pg_basebackup: error: FATAL: no pg_hba.conf entry for replication connection from host "10.101.100.11", user "admin", SSL off

I noticed the user "admin", and this is a bit strange since by default we use the user "replicator" for replication. Are you sure that no third-party environment variables are set on your servers?

now i rename username and trying to redeploy

masbolgankezderim commented 2 years ago

I change username to:

patroni_superuser_username: "masterator"
patroni_superuser_password: "masterator"  # please change password
patroni_replication_username: "replicator"
patroni_replication_password: "replicator"  # please change password

but it's still doesn't work (

masbolgankezderim commented 2 years ago 
[root@pgnode01 ~]# /usr/local/bin/patronictl -c /etc/patroni/patroni.yml list
+ Cluster: postgres-cluster (6997777838318695680) --+-----------+
| Member   | Host          | Role    | State   | TL | Lag in MB |
+----------+---------------+---------+---------+----+-----------+
| pgnode01 | 10.101.100.10 | Leader  | running |  1 |           |
| pgnode02 | 10.101.100.11 | Replica | stopped |    |   unknown |
| pgnode03 | 10.101.100.12 | Replica | stopped |    |   unknown |
+----------+---------------+---------+---------+----+-----------+
vitabaks commented 2 years ago

Thanks.

Please attach the Patroni and postgres logs from all nodes. and just in case, the result of executing the env command for the postgres user from all nodes of the cluster.

masbolgankezderim commented 2 years ago

master

[dar@pgnode01 ~]$ sudo tail /var/log/messages 
Aug 19 07:57:22 testing-vms-1 patroni: 2021-08-19 07:57:22,897 INFO: no action. I am (pgnode01) the leader with the lock
Aug 19 07:57:32 testing-vms-1 patroni: 2021-08-19 07:57:32,898 INFO: no action. I am (pgnode01) the leader with the lock
Aug 19 07:57:42 testing-vms-1 patroni: 2021-08-19 07:57:42,900 INFO: no action. I am (pgnode01) the leader with the lock
Aug 19 07:57:52 testing-vms-1 patroni: 2021-08-19 07:57:52,903 INFO: no action. I am (pgnode01) the leader with the lock
Aug 19 07:58:02 testing-vms-1 patroni: 2021-08-19 07:58:02,917 INFO: no action. I am (pgnode01) the leader with the lock
Aug 19 07:58:12 testing-vms-1 patroni: 2021-08-19 07:58:12,899 INFO: no action. I am (pgnode01) the leader with the lock
Aug 19 07:58:18 testing-vms-1 systemd: Created slice User Slice of dar.
Aug 19 07:58:18 testing-vms-1 systemd: Started Session 25 of user dar.
Aug 19 07:58:18 testing-vms-1 systemd-logind: New session 25 of user dar.
Aug 19 07:58:22 testing-vms-1 patroni: 2021-08-19 07:58:22,898 INFO: no action. I am (pgnode01) the leader with the lock

replica-1

[dar@pgnode02 ~]$ sudo tail /var/log/messages 
Aug 19 07:54:48 testing-vms-2 patroni: 2021-08-19 07:54:48,446 ERROR: Error when fetching backup: pg_basebackup exited with code=1
Aug 19 07:54:48 testing-vms-2 patroni: 2021-08-19 07:54:48,446 ERROR: Error when fetching backup: pg_basebackup exited with code=1
Aug 19 07:54:48 testing-vms-2 patroni: 2021-08-19 07:54:48,446 ERROR: failed to bootstrap from leader 'pgnode01'
Aug 19 07:54:48 testing-vms-2 patroni: 2021-08-19 07:54:48,446 INFO: Removing data directory: /var/lib/pgsql/13/data
Aug 19 07:54:48 testing-vms-2 rsyslogd: imjournal: journal reloaded... [v8.24.0-55.el7 try http://www.rsyslog.com/e/0 ]
Aug 19 07:54:53 testing-vms-2 patroni: 2021-08-19 07:54:53,357 INFO: Lock owner: pgnode01; I am pgnode02
Aug 19 07:54:53 testing-vms-2 patroni: 2021-08-19 07:54:53,368 INFO: trying to bootstrap from leader 'pgnode01'
Aug 19 07:54:53 testing-vms-2 patroni: pg_basebackup: error: FATAL:  no pg_hba.conf entry for replication connection from host "10.101.100.11", user "replicator", SSL off
Aug 19 07:54:53 testing-vms-2 patroni: 2021-08-19 07:54:53,397 ERROR: Error when fetching backup: pg_basebackup exited with code=1
Aug 19 07:54:53 testing-vms-2 patroni: 2021-08-19 07:54:53,397 WARNING: Trying again in 5 seconds

replica-2

[dar@pgnode03 ~]$ sudo tail /var/log/messages 
Aug 19 07:55:38 testing-vms-3 patroni: 2021-08-19 07:55:38,929 INFO: Removing data directory: /var/lib/pgsql/13/data
Aug 19 07:55:43 testing-vms-3 patroni: 2021-08-19 07:55:43,841 INFO: Lock owner: pgnode01; I am pgnode03
Aug 19 07:55:43 testing-vms-3 patroni: 2021-08-19 07:55:43,855 INFO: trying to bootstrap from leader 'pgnode01'
Aug 19 07:55:43 testing-vms-3 patroni: pg_basebackup: error: FATAL:  no pg_hba.conf entry for replication connection from host "10.101.100.12", user "replicator", SSL off
Aug 19 07:55:43 testing-vms-3 patroni: 2021-08-19 07:55:43,884 ERROR: Error when fetching backup: pg_basebackup exited with code=1
Aug 19 07:55:43 testing-vms-3 patroni: 2021-08-19 07:55:43,885 WARNING: Trying again in 5 seconds
Aug 19 07:55:48 testing-vms-3 patroni: pg_basebackup: error: FATAL:  no pg_hba.conf entry for replication connection from host "10.101.100.12", user "replicator", SSL off
Aug 19 07:55:48 testing-vms-3 patroni: 2021-08-19 07:55:48,919 ERROR: Error when fetching backup: pg_basebackup exited with code=1
Aug 19 07:55:48 testing-vms-3 patroni: 2021-08-19 07:55:48,921 ERROR: failed to bootstrap from leader 'pgnode01'
Aug 19 07:55:48 testing-vms-3 patroni: 2021-08-19 07:55:48,921 INFO: Removing data directory: /var/lib/pgsql/13/data

env for master node

[dar@pgnode01 ~]$ sudo -su postgres env
LC_PAPER=en_US.UTF-8
XDG_SESSION_ID=25
LC_ADDRESS=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
HOSTNAME=pgnode01
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
LC_NUMERIC=en_US.UTF-8
LC_ALL=en_US.utf-8
USER=postgres
LC_TELEPHONE=en_US.UTF-8
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
SUDO_USER=dar
SUDO_UID=1000
USERNAME=postgres
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAIL=/var/spool/mail/dar
LC_IDENTIFICATION=en_US.UTF-8
PWD=/home/dar
LANG=en_US.utf-8
LC_MEASUREMENT=en_US.UTF-8
SHLVL=1
SUDO_COMMAND=/bin/bash -c env
HOME=/var/lib/pgsql
LOGNAME=postgres
SUDO_GID=1000
LC_TIME=en_US.UTF-8
LC_NAME=en_US.UTF-8
_=/bin/env

replica-1

[dar@pgnode02 ~]$ sudo -su postgres env
LC_PAPER=en_US.UTF-8
XDG_SESSION_ID=21
LC_ADDRESS=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
HOSTNAME=pgnode02
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
LC_NUMERIC=en_US.UTF-8
LC_ALL=en_US.utf-8
USER=postgres
LC_TELEPHONE=en_US.UTF-8
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
SUDO_USER=dar
SUDO_UID=1000
USERNAME=postgres
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAIL=/var/spool/mail/dar
LC_IDENTIFICATION=en_US.UTF-8
PWD=/home/dar
LANG=en_US.utf-8
LC_MEASUREMENT=en_US.UTF-8
SHLVL=1
SUDO_COMMAND=/bin/bash -c env
HOME=/var/lib/pgsql
LOGNAME=postgres
SUDO_GID=1000
LC_TIME=en_US.UTF-8
LC_NAME=en_US.UTF-8
_=/bin/env

replica-2

[dar@pgnode03 ~]$ sudo -su postgres env
LC_PAPER=en_US.UTF-8
XDG_SESSION_ID=21
LC_ADDRESS=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
HOSTNAME=pgnode03
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
LC_NUMERIC=en_US.UTF-8
LC_ALL=en_US.utf-8
USER=postgres
LC_TELEPHONE=en_US.UTF-8
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
SUDO_USER=dar
SUDO_UID=1000
USERNAME=postgres
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAIL=/var/spool/mail/dar
LC_IDENTIFICATION=en_US.UTF-8
PWD=/home/dar
LANG=en_US.utf-8
LC_MEASUREMENT=en_US.UTF-8
SHLVL=1
SUDO_COMMAND=/bin/bash -c env
HOME=/var/lib/pgsql
LOGNAME=postgres
SUDO_GID=1000
LC_TIME=en_US.UTF-8
LC_NAME=en_US.UTF-8
_=/bin/env
vitabaks commented 2 years ago

pg_basebackup: error: FATAL: no pg_hba.conf entry for replication connection from host "10.101.100.11", user "replicator"

Very strange. Please attach pg_hba.conf from the master.

masbolgankezderim commented 2 years ago 
# TYPE      DATABASE                 USER                     ADDRESS                  METHOD
  local     all                      postgres                                          trust 
  local     all                      all                                               peer 
  host      all                      all                      127.0.0.1/32             md5 
  host      all                      all                      ::1/128                  md5 
  host      all                      all                      10.101.100.10/32         md5
  host      all                      all                      10.101.100.11/32         md5
  host      all                      all                      10.101.100.12/32         md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
  host      replication              replicator               localhost                trust
  host      replication              replicator               10.101.100.10/32         md5
  host      replication              replicator               10.101.100.11/32         md5
  host      replication              replicator               10.101.100.12/32         md5
vitabaks commented 2 years ago

hmmm

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *************************************************************************************************************************************************
changed: [10.101.100.10]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *************************************************************************************************************************************
ok: [10.101.100.10]

I see that the task "reload for apply the pg_hba.conf" was not performed. In tests, there was no such problem.

@AdilkhanKTR Try to apply the configuration on the Master and add replicas again.

on mater: SELECT pg_reload_conf();

then: ansible-playbook add_pgnode.yml