Provision of resources in a popular cloud providers for PostgreSQL cluster deployment

vitabaks / postgresql_cluster

PostgreSQL High-Availability Cluster (based on "Patroni" and DCS "etcd" or "consul"). Automating with Ansible.

MIT License

1.29k stars 352 forks source link

Provision of resources in a popular cloud providers for PostgreSQL cluster deployment #464

Open vitabaks opened 8 months ago

vitabaks commented 8 months ago

Issue: https://github.com/vitabaks/postgresql_cluster/issues/458

Provision of cloud resources (virtual machine + disk) to a popular cloud providers for PostgreSQL cluster deployment:
- [x] AWS
- [x] GCP
- [x] Azure
- [x] DigitalOcean
- [x] Hetzner Cloud
Create a Cloud Load Balancer (entry point for connecting to a database in a cluster)
- [x] AWS - Elastic Load Balancer (ELB)
- [ ] GCP (issue)
- [ ] Azure
- [x] DigitalOcean
- [x] Hetzner Cloud
Create a Bucket in the cloud and configure backups to this bucket (if 'pgbackrest_install' or 'wal_g_instal' is 'true')
- [x] AWS S3 Bucket
- [x] GCP Bucket
- [x] Azure Blob Storage
- [x] DigitalOcean Spaces
- [ ] Hetzner Cloud (?)

Role: cloud-resources

Examples:

AWS

export AWS_ACCESS_KEY_ID=********
export AWS_SECRET_ACCESS_KEY=********

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --extra-vars \
    "cloud_provider=aws \
     server_count=3 \
     server_location=us-east-2 \
     server_type=m5.xlarge \
     server_image=ami-05fb0b8c1424f266b \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Note: "ami-05fb0b8c1424f266b" is Ubuntu 22.04 LTS (from 2023-12-07)

GCP

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  ********
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider='gcp' \
     server_count=3 \
     server_location='us-central1' \
     server_type='n2-standard-4' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Azure

export AZURE_SUBSCRIPTION_ID=********
export AZURE_CLIENT_ID=********
export AZURE_SECRET=********
export AZURE_TENANT=********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider='azure' \
     server_count=3 \
     server_location='eastus' \
     server_type='Standard_D2s_v3' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Note: instead of 'server_image', use variables: azure_vm_image_offer, azure_vm_image_publisher, azure_vm_image_sku, azure_vm_image_version. Default: '0001-com-ubuntu-server-jammy', 'Canonical', '22_04-lts-gen2', 'latest'.

DigitalOcean

export DO_API_TOKEN=********

ansible-playbook deploy_pgcluster.yml \
  --user=root --extra-vars \
    "cloud_provider='digitalocean' \
     server_count=3 \
     server_location='nyc1' \
     server_type='g-4vcpu-16gb' \
     server_image='ubuntu-22-04-x64' \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Hetzner Cloud

export HCLOUD_API_TOKEN=********

ansible-playbook deploy_pgcluster.yml \
  --user=root --extra-vars \
    "cloud_provider='hetzner' \
     server_count=3 \
     server_location='ash' \
     server_type='CCX23' \
     server_image='ubuntu-22.04' \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\""

Additionally:

Docker image 'vitabaks/postgresql_cluster'
- includes repository code, ansible, and all necessary dependencies.
New roles:
- authorized-keys
- These SSH public keys will be added to the server's ~/.ssh/authorized_keys file.
- variable: ssh_public_keys (optional, default '')
- mount
- Configure mount points in/etc/fstab and mount the file system
- variables:
  - mount (optional, default '')
  - pg_data_mount_path (default '/pgdata')
  - pg_data_mount_fstype (default 'ext4')
- an empty disk will be automatically detected for cloud providers
- ZFS pool will be created if 'zfs' specified in the mount.fstype variable.
Automatically generate passwords for patroni_superuser_password, patroni_replication_password, pgbouncer_auth_password (if not defined)
- and print superuser password in “connection info” after deployment
- if the variable 'mask_password' is 'true' (default 'false'), do not print the superuser password in connection info
install perf, FlameGraph and postgres dbgsym/debuginfo packages
- variable install_perf, default 'false'
Extension Auto-Setup
- This feature simplifies the integration and configuration process for PostgreSQL third-party extensions. This automated approach eliminates the need for manual configuration, allowing users to seamlessly integrate PostgreSQL extensions. To activate the "Extension Auto-Setup", specify the enable_<extension_name>=true variable.
- List of extensions: timescaledb, citus, pg_repack, pg_cron, pgaudit, pgvector, postgis, pgrouting, pg_stat_kcache, pg_wait_sampling, pg_partman
Add the ability to execute pre and post-deploy command
- Variables: pre_deploy_command, post_deploy_command
- This can be a direct command, a bash script content, or a path to a script on the host.

Consider sponsoring the project via GitHub or Patreon

vitabaks commented 8 months ago

Test: AWS

export AWS_ACCESS_KEY_ID=**********
export AWS_SECRET_ACCESS_KEY=**********

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --private-key=$HOME/.ssh/Vitaliy.pem --extra-vars \
    "provision=aws \
     servers_count=3 \
     server_type=m5.large \
     server_image=ami-07d445a5585fce0a1 \
     server_location=us-east-2 \
     volume_size=60 \
     ssh_key_name=Vitaliy"

Result:

PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on AWS)] ****************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Gather information about default VPC] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] ******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] **************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] ****************************************************************************************************************************************************
changed: [localhost] => (item=postgres-cluster-pgnode01)
changed: [localhost] => (item=postgres-cluster-pgnode02)
changed: [localhost] => (item=postgres-cluster-pgnode03)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] *****************************************************************************************************************************************
ok: [localhost] => (item=18.223.205.113)
ok: [localhost] => (item=3.135.186.47)
ok: [localhost] => (item=3.144.48.18)

TASK [cloud-resources : Show EC2 instance info] ****************************************************************************************************************************************************************
ok: [localhost] => (item=18.223.205.113) => {
    "msg": [
        "instance ID is i-0215e0f507b416b43",
        "instance Image is ami-07d445a5585fce0a1",
        "instance Name is postgres-cluster-pgnode01",
        "instance Type is m5.large",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 18.223.205.113",
        "Private IP is 172.31.0.73"
    ]
}
ok: [localhost] => (item=3.135.186.47) => {
    "msg": [
        "instance ID is i-0733c8a9411959a41",
        "instance Image is ami-07d445a5585fce0a1",
        "instance Name is postgres-cluster-pgnode02",
        "instance Type is m5.large",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 3.135.186.47",
        "Private IP is 172.31.3.250"
    ]
}
ok: [localhost] => (item=3.144.48.18) => {
    "msg": [
        "instance ID is i-0eee7f56136793cf6",
        "instance Image is ami-07d445a5585fce0a1",
        "instance Name is postgres-cluster-pgnode03",
        "instance Type is m5.large",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 3.144.48.18",
        "Private IP is 172.31.7.185"
    ]
}

TASK [cloud-resources : Add host to "postgres_cluster", "master" groups (in-memory inventory)] *****************************************************************************************************************
ok: [localhost] => (item=18.223.205.113)

TASK [cloud-resources : Add host to "postgres_cluster", "replica" groups (in-memory inventory)] ****************************************************************************************************************
ok: [localhost] => (item=3.135.186.47)
ok: [localhost] => (item=3.144.48.18)

TASK [cloud-resources : Add host to "etcd_cluster" group (in-memory inventory)] ********************************************************************************************************************************
ok: [localhost] => (item=18.223.205.113)
ok: [localhost] => (item=3.135.186.47)
ok: [localhost] => (item=3.144.48.18)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, count: 1, cores: 1",
        "Disk space total": "77.75 GB",
        "Kernel": "5.15.0-1030-aws",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "m5.large",
        "Virtualization type": "kvm"
    }
}
ok: [172.31.3.250] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz, count: 1, cores: 1",
        "Disk space total": "77.75 GB",
        "Kernel": "5.15.0-1030-aws",
        "Memory": "7.47 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "m5.large",
        "Virtualization type": "kvm"
    }
}
ok: [172.31.7.185] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, count: 1, cores: 1",
        "Disk space total": "77.75 GB",
        "Kernel": "5.15.0-1030-aws",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "m5.large",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [172.31.0.73] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [172.31.0.73] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]
ok: [172.31.0.73]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [172.31.0.73] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=master)
ok: [172.31.3.250] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=postgres_cluster)
ok: [172.31.3.250] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=etcd_cluster)
ok: [172.31.3.250] => (item=replica)
ok: [172.31.7.185] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [172.31.3.250] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [172.31.7.185] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [172.31.0.73] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [172.31.7.185] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [172.31.3.250] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [172.31.3.250] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [172.31.0.73] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [172.31.3.250] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [172.31.7.185] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [172.31.0.73] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [172.31.3.250] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [172.31.7.185] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [172.31.0.73] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [172.31.3.250] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [172.31.7.185] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [172.31.0.73] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [172.31.3.250] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [172.31.7.185] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [172.31.0.73] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [172.31.3.250] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [172.31.7.185] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [172.31.0.73] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [172.31.3.250] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [172.31.7.185] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [172.31.0.73] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [172.31.3.250] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [172.31.7.185] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [172.31.0.73] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [172.31.3.250] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [172.31.0.73] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [172.31.7.185] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [172.31.0.73] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [172.31.3.250] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [172.31.7.185] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [172.31.3.250] => (item=etcd)
changed: [172.31.7.185] => (item=etcd)
changed: [172.31.0.73] => (item=etcd)
changed: [172.31.0.73] => (item=etcdctl)
changed: [172.31.3.250] => (item=etcdctl)
changed: [172.31.7.185] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "msg": "http://172.31.0.73:2379 is healthy: successfully committed proposal: took = 3.457788ms"
}
ok: [172.31.3.250] => {
    "msg": "http://172.31.3.250:2379 is healthy: successfully committed proposal: took = 3.650855ms"
}
ok: [172.31.7.185] => {
    "msg": "http://172.31.7.185:2379 is healthy: successfully committed proposal: took = 3.018372ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [172.31.0.73] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [172.31.3.250] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [172.31.7.185] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [172.31.3.250] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [172.31.7.185] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [172.31.0.73] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [172.31.3.250] => (item=['python3'])
ok: [172.31.7.185] => (item=['python3'])
ok: [172.31.0.73] => (item=['python3'])
ok: [172.31.3.250] => (item=python3)
ok: [172.31.7.185] => (item=python3)
ok: [172.31.0.73] => (item=python3)
changed: [172.31.7.185] => (item=python3-dev)
changed: [172.31.3.250] => (item=python3-dev)
changed: [172.31.0.73] => (item=python3-dev)
changed: [172.31.7.185] => (item=python3-psycopg2)
changed: [172.31.3.250] => (item=python3-psycopg2)
changed: [172.31.0.73] => (item=python3-psycopg2)
ok: [172.31.7.185] => (item=python3-setuptools)
ok: [172.31.3.250] => (item=python3-setuptools)
ok: [172.31.0.73] => (item=python3-setuptools)
changed: [172.31.3.250] => (item=python3-pip)
ok: [172.31.3.250] => (item=curl)
changed: [172.31.7.185] => (item=python3-pip)
ok: [172.31.3.250] => (item=less)
ok: [172.31.7.185] => (item=curl)
changed: [172.31.0.73] => (item=python3-pip)
ok: [172.31.3.250] => (item=sudo)
ok: [172.31.7.185] => (item=less)
ok: [172.31.0.73] => (item=curl)
ok: [172.31.3.250] => (item=vim)
ok: [172.31.7.185] => (item=sudo)
ok: [172.31.0.73] => (item=less)
ok: [172.31.3.250] => (item=gcc)
ok: [172.31.7.185] => (item=vim)
ok: [172.31.0.73] => (item=sudo)
ok: [172.31.7.185] => (item=gcc)
ok: [172.31.0.73] => (item=vim)
ok: [172.31.0.73] => (item=gcc)
changed: [172.31.3.250] => (item=jq)
ok: [172.31.3.250] => (item=iptables)
changed: [172.31.7.185] => (item=jq)
ok: [172.31.7.185] => (item=iptables)
changed: [172.31.0.73] => (item=jq)
ok: [172.31.0.73] => (item=iptables)
changed: [172.31.3.250] => (item=acl)
changed: [172.31.7.185] => (item=acl)
changed: [172.31.3.250] => (item=dnsutils)
changed: [172.31.0.73] => (item=acl)
changed: [172.31.7.185] => (item=dnsutils)
changed: [172.31.0.73] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.3.250]
changed: [172.31.0.73]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [172.31.3.250] => (item=postgresql-15)
ok: [172.31.3.250] => (item=postgresql-client-15)
changed: [172.31.0.73] => (item=postgresql-15)
ok: [172.31.3.250] => (item=postgresql-contrib-15)
ok: [172.31.0.73] => (item=postgresql-client-15)
ok: [172.31.0.73] => (item=postgresql-contrib-15)
changed: [172.31.7.185] => (item=postgresql-15)
ok: [172.31.7.185] => (item=postgresql-client-15)
ok: [172.31.7.185] => (item=postgresql-contrib-15)
changed: [172.31.0.73] => (item=postgresql-server-dev-15)
changed: [172.31.3.250] => (item=postgresql-server-dev-15)
changed: [172.31.7.185] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [172.31.3.250] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [172.31.0.73] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [172.31.7.185] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.0.73]
changed: [172.31.3.250]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.0.73]
changed: [172.31.3.250]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [172.31.0.73] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=master)
ok: [172.31.3.250] => (item=etcd_cluster)
ok: [172.31.0.73] => (item=postgres_cluster)
ok: [172.31.3.250] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=etcd_cluster)
ok: [172.31.3.250] => (item=replica)
ok: [172.31.7.185] => (item=postgres_cluster)
ok: [172.31.7.185] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [172.31.0.73] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [172.31.3.250] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [172.31.7.185] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [172.31.3.250] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [172.31.3.250] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [172.31.0.73] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [172.31.7.185] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [172.31.3.250] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [172.31.0.73] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [172.31.7.185] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [172.31.3.250] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [172.31.0.73] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [172.31.7.185] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [172.31.3.250] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [172.31.7.185] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [172.31.0.73] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [172.31.3.250] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [172.31.0.73] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [172.31.7.185] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [172.31.3.250] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [172.31.7.185] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [172.31.0.73] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [172.31.3.250] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [172.31.7.185] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [172.31.0.73] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [172.31.3.250] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [172.31.7.185] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [172.31.0.73] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [172.31.3.250] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [172.31.3.250] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [172.31.7.185] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [172.31.0.73] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [172.31.7.185] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [172.31.0.73] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [172.31.0.73] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [172.31.3.250] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [172.31.7.185] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [172.31.3.250] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [172.31.0.73] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [172.31.7.185] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [172.31.3.250] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [172.31.0.73] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [172.31.7.185] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [172.31.3.250] => (item=LANG=en_US.utf-8)
changed: [172.31.0.73] => (item=LANG=en_US.utf-8)
changed: [172.31.7.185] => (item=LANG=en_US.utf-8)
changed: [172.31.3.250] => (item=LANGUAGE=en_US.utf-8)
changed: [172.31.0.73] => (item=LANGUAGE=en_US.utf-8)
changed: [172.31.7.185] => (item=LANGUAGE=en_US.utf-8)
changed: [172.31.3.250] => (item=LC_ALL=en_US.utf-8)
changed: [172.31.0.73] => (item=LC_ALL=en_US.utf-8)
changed: [172.31.7.185] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [172.31.7.185]
ok: [172.31.3.250]
ok: [172.31.0.73]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.7.185]
ok: [172.31.3.250]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.7.185]
changed: [172.31.3.250]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [172.31.7.185]
changed: [172.31.0.73]
changed: [172.31.3.250]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [172.31.0.73] => (item=absent)
changed: [172.31.3.250] => (item=absent)
changed: [172.31.7.185] => (item=absent)
changed: [172.31.3.250] => (item=directory)
changed: [172.31.0.73] => (item=directory)
changed: [172.31.7.185] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [172.31.0.73]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [172.31.0.73]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [172.31.0.73]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.0.73]
changed: [172.31.7.185]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
ok: [172.31.3.250]
changed: [172.31.0.73]
ok: [172.31.7.185]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [172.31.7.185]
ok: [172.31.3.250]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [172.31.3.250]
changed: [172.31.7.185]
changed: [172.31.0.73]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [172.31.0.73]
changed: [172.31.3.250]
changed: [172.31.7.185]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [172.31.0.73] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [172.31.3.250]
ok: [172.31.0.73]
ok: [172.31.7.185]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [172.31.0.73]
ok: [172.31.3.250]
ok: [172.31.7.185]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [172.31.0.73]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: postgres-cluster -----+---------+-----------+----+-----------+",
        "| Member          | Host         | Role    | State     | TL | Lag in MB |",
        "+-----------------+--------------+---------+-----------+----+-----------+",
        "| ip-172-31-0-73  | 172.31.0.73  | Leader  | running   |  1 |           |",
        "| ip-172-31-3-250 | 172.31.3.250 | Replica | streaming |  1 |         0 |",
        "| ip-172-31-7-185 | 172.31.7.185 | Replica | streaming |  1 |         0 |",
        "+-----------------+--------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [172.31.0.73]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [172.31.0.73] => {
    "msg": [
        "+------------------------------------------------+",
        "address 172.31.0.73,172.31.3.250,172.31.7.185",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
172.31.0.73                : ok=104  changed=58   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
172.31.3.250               : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
172.31.7.185               : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=13   changed=2    unreachable=0    failed=0    skipped=18   rescued=0    ignored=0

passed

vitabaks commented 8 months ago

Test: GCP

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  *****
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='gcp' \
     servers_count=5 \
     server_type='n2-standard-4' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     server_location='us-central1' \
     volume_size=60 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:

PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on GCP)] ****************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'google-auth' dependency is present on controlling host] *******************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : GCP: Gather information about project] *************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_network_name] ********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Create or modify private firewall rule] *******************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify instance] ********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode04)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode05)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37)
ok: [localhost] => (item=34.134.213.137)
ok: [localhost] => (item=34.66.129.241)
ok: [localhost] => (item=35.192.39.18)
ok: [localhost] => (item=35.223.176.129)

TASK [cloud-resources : Show GCP instance info] ****************************************************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37) => {
    "msg": [
        "instance ID is 1335351888603344788",
        "instance Name is vitaliy-test-pgcluster-pgnode01",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 34.70.223.37",
        "Private IP is 10.128.15.213"
    ]
}
ok: [localhost] => (item=34.134.213.137) => {
    "msg": [
        "instance ID is 8329019901739626345",
        "instance Name is vitaliy-test-pgcluster-pgnode02",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 34.134.213.137",
        "Private IP is 10.128.15.216"
    ]
}
ok: [localhost] => (item=34.66.129.241) => {
    "msg": [
        "instance ID is 1438317652146220924",
        "instance Name is vitaliy-test-pgcluster-pgnode03",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 34.66.129.241",
        "Private IP is 10.128.15.217"
    ]
}
ok: [localhost] => (item=35.192.39.18) => {
    "msg": [
        "instance ID is 7150671383286607695",
        "instance Name is vitaliy-test-pgcluster-pgnode04",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 35.192.39.18",
        "Private IP is 10.128.15.218"
    ]
}
ok: [localhost] => (item=35.223.176.129) => {
    "msg": [
        "instance ID is 6973353061470793537",
        "instance Name is vitaliy-test-pgcluster-pgnode05",
        "instance Image is ubuntu-2204-lts",
        "instance Type is n2-standard-4",
        "Block Storage Size is 60 gigabytes",
        "Public IP is 35.223.176.129",
        "Private IP is 10.128.15.219"
    ]
}

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 34.70.223.37, private_ip: 10.128.15.213)
ok: [localhost] => (item=public_ip: 34.134.213.137, private_ip: 10.128.15.216)
ok: [localhost] => (item=public_ip: 34.66.129.241, private_ip: 10.128.15.217)
ok: [localhost] => (item=public_ip: 35.192.39.18, private_ip: 10.128.15.218)
ok: [localhost] => (item=public_ip: 35.223.176.129, private_ip: 10.128.15.219)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=34.134.213.137)
ok: [localhost] => (item=34.66.129.241)
ok: [localhost] => (item=35.192.39.18)
ok: [localhost] => (item=35.223.176.129)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=34.70.223.37)
ok: [localhost] => (item=34.134.213.137)
ok: [localhost] => (item=34.66.129.241)
ok: [localhost] => (item=35.192.39.18)
ok: [localhost] => (item=35.223.176.129)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]
ok: [10.128.15.216]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.216] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.217] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.218] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}
ok: [10.128.15.219] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU @ 2.80GHz, count: 1, cores: 2",
        "Disk space total": "78.0 GB",
        "Kernel": "6.2.0-1012-gcp",
        "Memory": "15.61 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Google Compute Engine",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.128.15.213] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.128.15.213] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.219]
ok: [10.128.15.218]
ok: [10.128.15.216]
ok: [10.128.15.217]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.216]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.216]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.219]
ok: [10.128.15.217]
ok: [10.128.15.218]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]
ok: [10.128.15.216]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.219]
ok: [10.128.15.213]
ok: [10.128.15.218]
ok: [10.128.15.217]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.128.15.218]
ok: [10.128.15.217]
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.219]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.128.15.213] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=master)
ok: [10.128.15.216] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=postgres_cluster)
ok: [10.128.15.216] => (item=postgres_cluster)
ok: [10.128.15.217] => (item=etcd_cluster)
ok: [10.128.15.216] => (item=replica)
ok: [10.128.15.217] => (item=postgres_cluster)
ok: [10.128.15.218] => (item=etcd_cluster)
ok: [10.128.15.217] => (item=replica)
ok: [10.128.15.218] => (item=postgres_cluster)
ok: [10.128.15.219] => (item=etcd_cluster)
ok: [10.128.15.218] => (item=replica)
ok: [10.128.15.219] => (item=postgres_cluster)
ok: [10.128.15.219] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.219] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.216] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.217] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.218] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.128.15.213] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.213] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.219] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.217] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.213] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.218] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.217] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.213] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.219] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.218] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.217] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.216] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.128.15.219] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.213] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.218] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.219] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.213] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.218] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.216] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.128.15.213] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.219] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.218] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.213] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.219] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.216] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.128.15.217] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.218] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.218] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.218] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.216] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.219] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.128.15.213] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.219] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.216] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.218] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.218] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.128.15.216] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.128.15.216] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.218]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.128.15.219] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.128.15.217] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.128.15.213] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.128.15.216] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.128.15.218] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.219]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.128.15.213] => (item=etcd)
changed: [10.128.15.217] => (item=etcd)
changed: [10.128.15.216] => (item=etcd)
changed: [10.128.15.218] => (item=etcd)
changed: [10.128.15.219] => (item=etcd)
changed: [10.128.15.213] => (item=etcdctl)
changed: [10.128.15.217] => (item=etcdctl)
changed: [10.128.15.216] => (item=etcdctl)
changed: [10.128.15.218] => (item=etcdctl)
changed: [10.128.15.219] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.218]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.216]
changed: [10.128.15.218]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]
changed: [10.128.15.219]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.217]
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.218]
ok: [10.128.15.217]
ok: [10.128.15.216]
ok: [10.128.15.219]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "msg": "http://10.128.15.213:2379 is healthy: successfully committed proposal: took = 4.548685ms"
}
ok: [10.128.15.216] => {
    "msg": "http://10.128.15.216:2379 is healthy: successfully committed proposal: took = 3.359365ms"
}
ok: [10.128.15.217] => {
    "msg": "http://10.128.15.217:2379 is healthy: successfully committed proposal: took = 4.280164ms"
}
ok: [10.128.15.218] => {
    "msg": "http://10.128.15.218:2379 is healthy: successfully committed proposal: took = 3.928711ms"
}
ok: [10.128.15.219] => {
    "msg": "http://10.128.15.219:2379 is healthy: successfully committed proposal: took = 3.204371ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.217]
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.217] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.219] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.218] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.128.15.216] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.217] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.218] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.216] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.128.15.219] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.128.15.213] => (item=['python3'])
ok: [10.128.15.217] => (item=['python3'])
ok: [10.128.15.216] => (item=['python3'])
ok: [10.128.15.219] => (item=['python3'])
ok: [10.128.15.218] => (item=['python3'])
ok: [10.128.15.213] => (item=python3)
ok: [10.128.15.217] => (item=python3)
ok: [10.128.15.216] => (item=python3)
ok: [10.128.15.219] => (item=python3)
ok: [10.128.15.218] => (item=python3)
changed: [10.128.15.217] => (item=python3-dev)
changed: [10.128.15.219] => (item=python3-dev)
changed: [10.128.15.213] => (item=python3-dev)
changed: [10.128.15.218] => (item=python3-dev)
changed: [10.128.15.216] => (item=python3-dev)
changed: [10.128.15.217] => (item=python3-psycopg2)
changed: [10.128.15.219] => (item=python3-psycopg2)
ok: [10.128.15.217] => (item=python3-setuptools)
changed: [10.128.15.218] => (item=python3-psycopg2)
changed: [10.128.15.216] => (item=python3-psycopg2)
changed: [10.128.15.213] => (item=python3-psycopg2)
ok: [10.128.15.219] => (item=python3-setuptools)
ok: [10.128.15.218] => (item=python3-setuptools)
ok: [10.128.15.213] => (item=python3-setuptools)
ok: [10.128.15.216] => (item=python3-setuptools)
changed: [10.128.15.217] => (item=python3-pip)
changed: [10.128.15.219] => (item=python3-pip)
changed: [10.128.15.218] => (item=python3-pip)
changed: [10.128.15.213] => (item=python3-pip)
ok: [10.128.15.217] => (item=curl)
ok: [10.128.15.219] => (item=curl)
ok: [10.128.15.218] => (item=curl)
changed: [10.128.15.216] => (item=python3-pip)
ok: [10.128.15.213] => (item=curl)
ok: [10.128.15.217] => (item=less)
ok: [10.128.15.219] => (item=less)
ok: [10.128.15.218] => (item=less)
ok: [10.128.15.213] => (item=less)
ok: [10.128.15.216] => (item=curl)
ok: [10.128.15.217] => (item=sudo)
ok: [10.128.15.219] => (item=sudo)
ok: [10.128.15.218] => (item=sudo)
ok: [10.128.15.213] => (item=sudo)
ok: [10.128.15.216] => (item=less)
ok: [10.128.15.217] => (item=vim)
ok: [10.128.15.219] => (item=vim)
ok: [10.128.15.218] => (item=vim)
ok: [10.128.15.213] => (item=vim)
ok: [10.128.15.216] => (item=sudo)
ok: [10.128.15.217] => (item=gcc)
ok: [10.128.15.219] => (item=gcc)
ok: [10.128.15.218] => (item=gcc)
ok: [10.128.15.213] => (item=gcc)
ok: [10.128.15.216] => (item=vim)
ok: [10.128.15.216] => (item=gcc)
changed: [10.128.15.219] => (item=jq)
changed: [10.128.15.217] => (item=jq)
changed: [10.128.15.218] => (item=jq)
changed: [10.128.15.213] => (item=jq)
ok: [10.128.15.219] => (item=iptables)
ok: [10.128.15.217] => (item=iptables)
ok: [10.128.15.218] => (item=iptables)
ok: [10.128.15.213] => (item=iptables)
changed: [10.128.15.216] => (item=jq)
ok: [10.128.15.216] => (item=iptables)
changed: [10.128.15.219] => (item=acl)
changed: [10.128.15.217] => (item=acl)
changed: [10.128.15.213] => (item=acl)
changed: [10.128.15.218] => (item=acl)
changed: [10.128.15.216] => (item=acl)
changed: [10.128.15.219] => (item=dnsutils)
changed: [10.128.15.217] => (item=dnsutils)
changed: [10.128.15.213] => (item=dnsutils)
changed: [10.128.15.218] => (item=dnsutils)
changed: [10.128.15.216] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.216]
changed: [10.128.15.217]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.128.15.213] => (item=postgresql-15)
changed: [10.128.15.216] => (item=postgresql-15)
changed: [10.128.15.217] => (item=postgresql-15)
changed: [10.128.15.218] => (item=postgresql-15)
ok: [10.128.15.213] => (item=postgresql-client-15)
ok: [10.128.15.216] => (item=postgresql-client-15)
changed: [10.128.15.219] => (item=postgresql-15)
ok: [10.128.15.217] => (item=postgresql-client-15)
ok: [10.128.15.218] => (item=postgresql-client-15)
ok: [10.128.15.213] => (item=postgresql-contrib-15)
ok: [10.128.15.216] => (item=postgresql-contrib-15)
ok: [10.128.15.219] => (item=postgresql-client-15)
ok: [10.128.15.217] => (item=postgresql-contrib-15)
ok: [10.128.15.218] => (item=postgresql-contrib-15)
ok: [10.128.15.219] => (item=postgresql-contrib-15)
changed: [10.128.15.216] => (item=postgresql-server-dev-15)
changed: [10.128.15.213] => (item=postgresql-server-dev-15)
changed: [10.128.15.218] => (item=postgresql-server-dev-15)
changed: [10.128.15.219] => (item=postgresql-server-dev-15)
changed: [10.128.15.217] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.128.15.213] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.216] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.219] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.218] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.128.15.217] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.128.15.213] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=master)
ok: [10.128.15.216] => (item=etcd_cluster)
ok: [10.128.15.213] => (item=postgres_cluster)
ok: [10.128.15.216] => (item=postgres_cluster)
ok: [10.128.15.217] => (item=etcd_cluster)
ok: [10.128.15.216] => (item=replica)
ok: [10.128.15.217] => (item=postgres_cluster)
ok: [10.128.15.218] => (item=etcd_cluster)
ok: [10.128.15.217] => (item=replica)
ok: [10.128.15.218] => (item=postgres_cluster)
ok: [10.128.15.218] => (item=replica)
ok: [10.128.15.219] => (item=etcd_cluster)
ok: [10.128.15.219] => (item=postgres_cluster)
ok: [10.128.15.219] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.128.15.213] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.216] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.217] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.218] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.219] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.128.15.213] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.217] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.128.15.213] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.216] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.217] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.218] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.219] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.128.15.213] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.216] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.217] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.219] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.218] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.128.15.213] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.216] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.217] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.219] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.213] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.216] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.217] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.213] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.213] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.217] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.219] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.213] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.218] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.128.15.217] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.217] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.218] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.213] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.219] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.218] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.216] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.217] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.213] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.213] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.128.15.216] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.217] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.216] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.128.15.217] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.219] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.128.15.219] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.128.15.218] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.128.15.218] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.128.15.218] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.217]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.128.15.213] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.218] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.217] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.216] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.213] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.218] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.217] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.216] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.128.15.219] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.128.15.219] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.128.15.213] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.218] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.216] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.217] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.128.15.219] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.128.15.213] => (item=LANG=en_US.utf-8)
changed: [10.128.15.218] => (item=LANG=en_US.utf-8)
changed: [10.128.15.216] => (item=LANG=en_US.utf-8)
changed: [10.128.15.217] => (item=LANG=en_US.utf-8)
changed: [10.128.15.213] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.219] => (item=LANG=en_US.utf-8)
changed: [10.128.15.218] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.216] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.217] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.213] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.218] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.216] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.217] => (item=LC_ALL=en_US.utf-8)
changed: [10.128.15.219] => (item=LANGUAGE=en_US.utf-8)
changed: [10.128.15.219] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.128.15.219]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.213]
changed: [10.128.15.218]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]
changed: [10.128.15.219]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.217]
changed: [10.128.15.213]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.217]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.213]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.213]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.219]
changed: [10.128.15.218]
changed: [10.128.15.217]
changed: [10.128.15.213]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]
changed: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.213]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.217]
ok: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.217]
changed: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [10.128.15.216] => (item=absent)
changed: [10.128.15.218] => (item=absent)
changed: [10.128.15.219] => (item=absent)
changed: [10.128.15.216] => (item=directory)
changed: [10.128.15.218] => (item=directory)
changed: [10.128.15.219] => (item=directory)
changed: [10.128.15.213] => (item=absent)
changed: [10.128.15.217] => (item=absent)
changed: [10.128.15.213] => (item=directory)
changed: [10.128.15.217] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.128.15.213]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.128.15.213]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.128.15.213]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.128.15.213]
changed: [10.128.15.218]
changed: [10.128.15.216]
changed: [10.128.15.217]
changed: [10.128.15.219]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
changed: [10.128.15.213]
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.217]
ok: [10.128.15.219]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.217]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.128.15.219]
ok: [10.128.15.218]
ok: [10.128.15.216]
ok: [10.128.15.217]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.217]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.219]
ok: [10.128.15.218]
ok: [10.128.15.213]
ok: [10.128.15.217]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.128.15.216]
changed: [10.128.15.218]
changed: [10.128.15.219]
changed: [10.128.15.213]
changed: [10.128.15.217]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.128.15.213] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.213]
ok: [10.128.15.217]
ok: [10.128.15.219]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.128.15.216]
ok: [10.128.15.218]
ok: [10.128.15.219]
ok: [10.128.15.217]
ok: [10.128.15.213]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.128.15.213]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +---------------+---------+-----------+----+-----------+",
        "| Member                          | Host          | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+---------------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.128.15.213 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.128.15.216 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.128.15.217 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode04 | 10.128.15.218 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode05 | 10.128.15.219 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+---------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.128.15.213]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.128.15.213] => {
    "msg": [
        "+------------------------------------------------+",
        "address 10.128.15.213,10.128.15.216,10.128.15.217,10.128.15.218,10.128.15.219",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.128.15.213              : ok=104  changed=58   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
10.128.15.216              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.128.15.217              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.128.15.218              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.128.15.219              : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=13   changed=2    unreachable=0    failed=0    skipped=41   rescued=0    ignored=0

passed

vitabaks commented 8 months ago

Test: DigitalOcean

export DO_API_TOKEN=********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='digitalocean' \
     servers_count=3 \
     server_type='g-4vcpu-16gb' \
     server_image='ubuntu-22-04-x64' \
     server_location='nyc1' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:

PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on DIGITALOCEAN)] *******************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get Fingerprint for SSH key 'vitaliy'] ***********************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] ********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] *****************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitaliy-test-pgcluster'] ***********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify SSH firewall rule] **************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify private firewall rule] **********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] ************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Set variable: droplet_result] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] ******************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] *****************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240)
ok: [localhost] => (item=167.99.228.9)
ok: [localhost] => (item=159.89.84.26)

TASK [cloud-resources : Show Droplet info] *********************************************************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240) => {
    "msg": [
        "ID: 374097982",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-4vcpu-16gb",
        "Volume Size: 100 GB",
        "Public IP: 198.199.90.240",
        "Private IP: 10.116.0.2"
    ]
}
ok: [localhost] => (item=167.99.228.9) => {
    "msg": [
        "ID: 374098039",
        "Name: vitaliy-test-pgcluster-pgnode02",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-4vcpu-16gb",
        "Volume Size: 100 GB",
        "Public IP: 167.99.228.9",
        "Private IP: 10.116.0.3"
    ]
}
ok: [localhost] => (item=159.89.84.26) => {
    "msg": [
        "ID: 374098074",
        "Name: vitaliy-test-pgcluster-pgnode03",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-4vcpu-16gb",
        "Volume Size: 100 GB",
        "Public IP: 159.89.84.26",
        "Private IP: 10.116.0.4"
    ]
}

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 198.199.90.240, private_ip: 10.116.0.2)
ok: [localhost] => (item=public_ip: 167.99.228.9, private_ip: 10.116.0.3)
ok: [localhost] => (item=public_ip: 159.89.84.26, private_ip: 10.116.0.4)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=167.99.228.9)
ok: [localhost] => (item=159.89.84.26)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=198.199.90.240)
ok: [localhost] => (item=167.99.228.9)
ok: [localhost] => (item=159.89.84.26)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz, count: 1, cores: 4",
        "Disk space total": "48.6 GB",
        "Kernel": "5.15.0-67-generic",
        "Memory": "15.63 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Droplet",
        "Virtualization type": "kvm"
    }
}
ok: [10.116.0.3] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz, count: 1, cores: 4",
        "Disk space total": "48.6 GB",
        "Kernel": "5.15.0-67-generic",
        "Memory": "15.63 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Droplet",
        "Virtualization type": "kvm"
    }
}
ok: [10.116.0.4] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz, count: 1, cores: 4",
        "Disk space total": "48.6 GB",
        "Kernel": "5.15.0-67-generic",
        "Memory": "15.63 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Droplet",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.116.0.2] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.116.0.2] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.2]
ok: [10.116.0.3]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.116.0.2] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=master)
ok: [10.116.0.3] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=postgres_cluster)
ok: [10.116.0.3] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=etcd_cluster)
ok: [10.116.0.3] => (item=replica)
ok: [10.116.0.4] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.116.0.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.116.0.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.116.0.4] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.116.0.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.116.0.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.116.0.2] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.116.0.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.116.0.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.116.0.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.116.0.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.116.0.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.116.0.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.116.0.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.116.0.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.116.0.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.116.0.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.116.0.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.116.0.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.116.0.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.116.0.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.116.0.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.116.0.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.116.0.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.116.0.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.116.0.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.116.0.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.116.0.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.116.0.4] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.116.0.3] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.116.0.2] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.116.0.4] => (item=etcd)
changed: [10.116.0.3] => (item=etcd)
changed: [10.116.0.2] => (item=etcd)
changed: [10.116.0.4] => (item=etcdctl)
changed: [10.116.0.2] => (item=etcdctl)
changed: [10.116.0.3] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "msg": "http://10.116.0.2:2379 is healthy: successfully committed proposal: took = 5.598689ms"
}
ok: [10.116.0.3] => {
    "msg": "http://10.116.0.3:2379 is healthy: successfully committed proposal: took = 4.467634ms"
}
ok: [10.116.0.4] => {
    "msg": "http://10.116.0.4:2379 is healthy: successfully committed proposal: took = 6.142228ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.116.0.3] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.116.0.4] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.116.0.2] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.116.0.4] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.116.0.3] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.116.0.2] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.116.0.2] => (item=['python3'])
ok: [10.116.0.4] => (item=['python3'])
ok: [10.116.0.3] => (item=['python3'])
ok: [10.116.0.4] => (item=python3)
ok: [10.116.0.3] => (item=python3)
ok: [10.116.0.2] => (item=python3)
changed: [10.116.0.4] => (item=python3-dev)
changed: [10.116.0.3] => (item=python3-dev)
changed: [10.116.0.4] => (item=python3-psycopg2)
changed: [10.116.0.2] => (item=python3-dev)
ok: [10.116.0.4] => (item=python3-setuptools)
changed: [10.116.0.3] => (item=python3-psycopg2)
ok: [10.116.0.3] => (item=python3-setuptools)
changed: [10.116.0.2] => (item=python3-psycopg2)
changed: [10.116.0.4] => (item=python3-pip)
ok: [10.116.0.2] => (item=python3-setuptools)
ok: [10.116.0.4] => (item=curl)
ok: [10.116.0.4] => (item=less)
changed: [10.116.0.3] => (item=python3-pip)
ok: [10.116.0.4] => (item=sudo)
ok: [10.116.0.3] => (item=curl)
ok: [10.116.0.4] => (item=vim)
ok: [10.116.0.3] => (item=less)
ok: [10.116.0.4] => (item=gcc)
ok: [10.116.0.3] => (item=sudo)
ok: [10.116.0.3] => (item=vim)
ok: [10.116.0.3] => (item=gcc)
changed: [10.116.0.4] => (item=jq)
ok: [10.116.0.4] => (item=iptables)
changed: [10.116.0.2] => (item=python3-pip)
ok: [10.116.0.2] => (item=curl)
changed: [10.116.0.3] => (item=jq)
ok: [10.116.0.2] => (item=less)
ok: [10.116.0.3] => (item=iptables)
changed: [10.116.0.4] => (item=acl)
ok: [10.116.0.2] => (item=sudo)
ok: [10.116.0.2] => (item=vim)
changed: [10.116.0.4] => (item=dnsutils)
changed: [10.116.0.3] => (item=acl)
ok: [10.116.0.2] => (item=gcc)
changed: [10.116.0.3] => (item=dnsutils)
changed: [10.116.0.2] => (item=jq)
ok: [10.116.0.2] => (item=iptables)
changed: [10.116.0.2] => (item=acl)
changed: [10.116.0.2] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.116.0.4] => (item=postgresql-15)
changed: [10.116.0.3] => (item=postgresql-15)
ok: [10.116.0.4] => (item=postgresql-client-15)
changed: [10.116.0.2] => (item=postgresql-15)
ok: [10.116.0.3] => (item=postgresql-client-15)
ok: [10.116.0.4] => (item=postgresql-contrib-15)
ok: [10.116.0.2] => (item=postgresql-client-15)
ok: [10.116.0.3] => (item=postgresql-contrib-15)
ok: [10.116.0.2] => (item=postgresql-contrib-15)
changed: [10.116.0.4] => (item=postgresql-server-dev-15)
changed: [10.116.0.3] => (item=postgresql-server-dev-15)
changed: [10.116.0.2] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.116.0.4] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.116.0.2] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.116.0.3] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]
changed: [10.116.0.2]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.116.0.2] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=master)
ok: [10.116.0.3] => (item=etcd_cluster)
ok: [10.116.0.2] => (item=postgres_cluster)
ok: [10.116.0.3] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=etcd_cluster)
ok: [10.116.0.3] => (item=replica)
ok: [10.116.0.4] => (item=postgres_cluster)
ok: [10.116.0.4] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.116.0.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.116.0.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.116.0.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.116.0.4] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.116.0.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.116.0.3] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.116.0.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.116.0.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.116.0.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.116.0.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.116.0.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.116.0.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.116.0.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.116.0.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.116.0.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.116.0.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.116.0.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.116.0.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.116.0.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.116.0.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.116.0.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.116.0.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.116.0.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.116.0.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.116.0.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.116.0.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.116.0.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.116.0.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.116.0.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.2]
changed: [10.116.0.4]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.116.0.2] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.116.0.4] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.116.0.3] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.116.0.2] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.116.0.4] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.116.0.3] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.116.0.4] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.116.0.3] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.116.0.2] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.116.0.4] => (item=LANG=en_US.utf-8)
changed: [10.116.0.2] => (item=LANG=en_US.utf-8)
changed: [10.116.0.3] => (item=LANG=en_US.utf-8)
changed: [10.116.0.4] => (item=LANGUAGE=en_US.utf-8)
changed: [10.116.0.2] => (item=LANGUAGE=en_US.utf-8)
changed: [10.116.0.3] => (item=LANGUAGE=en_US.utf-8)
changed: [10.116.0.4] => (item=LC_ALL=en_US.utf-8)
changed: [10.116.0.2] => (item=LC_ALL=en_US.utf-8)
changed: [10.116.0.3] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.4]
changed: [10.116.0.3]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.2]
changed: [10.116.0.4]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [10.116.0.2]
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.116.0.2]
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.2]
changed: [10.116.0.3]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [10.116.0.4] => (item=absent)
changed: [10.116.0.3] => (item=absent)
changed: [10.116.0.4] => (item=directory)
changed: [10.116.0.3] => (item=directory)
changed: [10.116.0.2] => (item=absent)
changed: [10.116.0.2] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.116.0.2]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.116.0.2]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.116.0.4]
changed: [10.116.0.3]
changed: [10.116.0.2]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
changed: [10.116.0.2]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]
changed: [10.116.0.2]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.116.0.3]
changed: [10.116.0.4]
changed: [10.116.0.2]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.116.0.2] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.116.0.3]
ok: [10.116.0.4]
ok: [10.116.0.2]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.116.0.4]
ok: [10.116.0.3]
ok: [10.116.0.2]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.116.0.2]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +------------+---------+-----------+----+-----------+",
        "| Member                          | Host       | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+------------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "msg": [
        "+------------------------------------------------+",
        "address 10.116.0.2,10.116.0.3,10.116.0.4",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.116.0.2                 : ok=104  changed=56   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
10.116.0.3                 : ok=89   changed=53   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.116.0.4                 : ok=89   changed=53   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=20   changed=5    unreachable=0    failed=0    skipped=70   rescued=0    ignored=0

passed

vitabaks commented 8 months ago

Test: Hetzner

export HCLOUD_API_TOKEN=******

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='hetzner' \
     servers_count=3 \
     server_type='CCX13' \
     server_image='ubuntu-22.04' \
     server_location='ash' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:


PLAY [Provision of cloud resources (virtual machine + disk) for PostgreSQL HA Cluster (on HETZNER)] ************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'hcloud' dependency is present on controlling host] ************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Hetzner Cloud: Gather information about SSH key 'vitaliy'] *****************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: ssh_key_names] ***********************************************************************************************************************************************************
ok: [localhost] => (item=None)
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Gather information about network zones] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Extract network zone for server_location] *******************************************************************************************************************************
ok: [localhost] => (item=network_zone: us-east)

TASK [cloud-resources : Hetzner Cloud: Gather information about networks] **************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a network 'postgres-cluster-network'] ****************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a subnetwork in the network 'postgres-cluster-network'] **********************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: server_network] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify SSH firewall] ******************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify private firewall] **************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Gather information about firewalls] *************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Extract firewall names for 'vitaliy-test-pgcluster'] ********************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify server] ************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Add server to network 'postgres-cluster-network'] ***********************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Create or modify volume] ************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76)
ok: [localhost] => (item=5.161.63.15)
ok: [localhost] => (item=5.161.229.14)

TASK [cloud-resources : Show Server info] **********************************************************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76) => {
    "msg": [
        "ID: 37095542",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.228.76",
        "Private IP: 10.0.1.1"
    ]
}
ok: [localhost] => (item=5.161.63.15) => {
    "msg": [
        "ID: 37095550",
        "Name: vitaliy-test-pgcluster-pgnode02",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.63.15",
        "Private IP: 10.0.1.2"
    ]
}
ok: [localhost] => (item=5.161.229.14) => {
    "msg": [
        "ID: 37095551",
        "Name: vitaliy-test-pgcluster-pgnode03",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.229.14",
        "Private IP: 10.0.1.3"
    ]
}

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 5.161.228.76, private_ip: 10.0.1.1)
ok: [localhost] => (item=public_ip: 5.161.63.15, private_ip: 10.0.1.2)
ok: [localhost] => (item=public_ip: 5.161.229.14, private_ip: 10.0.1.3)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=5.161.63.15)
ok: [localhost] => (item=5.161.229.14)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=5.161.228.76)
ok: [localhost] => (item=5.161.63.15)
ok: [localhost] => (item=5.161.229.14)

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "AMD EPYC Processor, count: 1, cores: 1",
        "Disk space total": "75.04 GB",
        "Kernel": "5.15.0-79-generic",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "vServer",
        "Virtualization type": "kvm"
    }
}
ok: [10.0.1.2] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "AMD EPYC Processor, count: 1, cores: 1",
        "Disk space total": "75.04 GB",
        "Kernel": "5.15.0-79-generic",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "vServer",
        "Virtualization type": "kvm"
    }
}
ok: [10.0.1.3] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "AMD EPYC Processor, count: 1, cores: 1",
        "Disk space total": "75.04 GB",
        "Kernel": "5.15.0-79-generic",
        "Memory": "7.56 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "vServer",
        "Virtualization type": "kvm"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.0.1.1] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.0.1.1] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ****************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [10.0.1.3]
changed: [10.0.1.1]
changed: [10.0.1.2]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.0.1.3]
ok: [10.0.1.2]
ok: [10.0.1.1]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.1] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=master)
ok: [10.0.1.2] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=postgres_cluster)
ok: [10.0.1.2] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=etcd_cluster)
ok: [10.0.1.2] => (item=replica)
ok: [10.0.1.3] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.0.1.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.1] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.1] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.1] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.1] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.1] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.1] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.1] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.1] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.1] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.1] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.1] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.0.1.1] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.0.1.2] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.0.1.3] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.0.1.1] => (item=etcd)
changed: [10.0.1.3] => (item=etcd)
changed: [10.0.1.2] => (item=etcd)
changed: [10.0.1.1] => (item=etcdctl)
changed: [10.0.1.3] => (item=etcdctl)
changed: [10.0.1.2] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "msg": "http://10.0.1.1:2379 is healthy: successfully committed proposal: took = 2.963209ms"
}
ok: [10.0.1.2] => {
    "msg": "http://10.0.1.2:2379 is healthy: successfully committed proposal: took = 3.411725ms"
}
ok: [10.0.1.3] => {
    "msg": "http://10.0.1.3:2379 is healthy: successfully committed proposal: took = 2.991831ms"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.0.1.3] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.1] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.2] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.0.1.2] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.3] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.1] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.0.1.1] => (item=['python3'])
ok: [10.0.1.3] => (item=['python3'])
ok: [10.0.1.2] => (item=['python3'])
ok: [10.0.1.1] => (item=python3)
ok: [10.0.1.3] => (item=python3)
ok: [10.0.1.2] => (item=python3)
changed: [10.0.1.3] => (item=python3-dev)
changed: [10.0.1.2] => (item=python3-dev)
changed: [10.0.1.1] => (item=python3-dev)
changed: [10.0.1.3] => (item=python3-psycopg2)
changed: [10.0.1.2] => (item=python3-psycopg2)
changed: [10.0.1.1] => (item=python3-psycopg2)
ok: [10.0.1.3] => (item=python3-setuptools)
ok: [10.0.1.1] => (item=python3-setuptools)
ok: [10.0.1.2] => (item=python3-setuptools)
changed: [10.0.1.3] => (item=python3-pip)
ok: [10.0.1.3] => (item=curl)
changed: [10.0.1.1] => (item=python3-pip)
ok: [10.0.1.3] => (item=less)
ok: [10.0.1.1] => (item=curl)
changed: [10.0.1.2] => (item=python3-pip)
ok: [10.0.1.3] => (item=sudo)
ok: [10.0.1.1] => (item=less)
ok: [10.0.1.2] => (item=curl)
ok: [10.0.1.3] => (item=vim)
ok: [10.0.1.1] => (item=sudo)
ok: [10.0.1.2] => (item=less)
ok: [10.0.1.3] => (item=gcc)
ok: [10.0.1.1] => (item=vim)
ok: [10.0.1.2] => (item=sudo)
ok: [10.0.1.1] => (item=gcc)
ok: [10.0.1.2] => (item=vim)
changed: [10.0.1.3] => (item=jq)
ok: [10.0.1.2] => (item=gcc)
ok: [10.0.1.3] => (item=iptables)
changed: [10.0.1.1] => (item=jq)
ok: [10.0.1.3] => (item=acl)
changed: [10.0.1.2] => (item=jq)
ok: [10.0.1.1] => (item=iptables)
ok: [10.0.1.2] => (item=iptables)
ok: [10.0.1.1] => (item=acl)
changed: [10.0.1.3] => (item=dnsutils)
ok: [10.0.1.2] => (item=acl)
changed: [10.0.1.1] => (item=dnsutils)
changed: [10.0.1.2] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.0.1.1] => (item=postgresql-15)
changed: [10.0.1.2] => (item=postgresql-15)
changed: [10.0.1.3] => (item=postgresql-15)
ok: [10.0.1.1] => (item=postgresql-client-15)
ok: [10.0.1.2] => (item=postgresql-client-15)
ok: [10.0.1.3] => (item=postgresql-client-15)
ok: [10.0.1.1] => (item=postgresql-contrib-15)
ok: [10.0.1.2] => (item=postgresql-contrib-15)
ok: [10.0.1.3] => (item=postgresql-contrib-15)
changed: [10.0.1.2] => (item=postgresql-server-dev-15)
changed: [10.0.1.3] => (item=postgresql-server-dev-15)
changed: [10.0.1.1] => (item=postgresql-server-dev-15)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.0.1.1] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.3] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.2] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.1] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=master)
ok: [10.0.1.2] => (item=etcd_cluster)
ok: [10.0.1.1] => (item=postgres_cluster)
ok: [10.0.1.2] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=etcd_cluster)
ok: [10.0.1.2] => (item=replica)
ok: [10.0.1.3] => (item=postgres_cluster)
ok: [10.0.1.3] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.0.1.1] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.2] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.3] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.1] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.1] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.2] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.3] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.1] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.2] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.3] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.1] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.2] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.3] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.1] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.2] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.3] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.1] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.2] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.3] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.1] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.2] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.3] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.1] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.2] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.3] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.1] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.3] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.2] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.1] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.3] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.2] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.1] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.3] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.2] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.1]
changed: [10.0.1.3]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.0.1.1] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.3] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.2] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.1] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.3] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.2] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.0.1.1] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.3] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.2] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.0.1.1] => (item=LANG=en_US.utf-8)
changed: [10.0.1.2] => (item=LANG=en_US.utf-8)
changed: [10.0.1.3] => (item=LANG=en_US.utf-8)
changed: [10.0.1.1] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.2] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.3] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.1] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.3] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.2] => (item=LC_ALL=en_US.utf-8)

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.2]
ok: [10.0.1.1]
ok: [10.0.1.3]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.1]
changed: [10.0.1.3]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]
changed: [10.0.1.1]

TASK [patroni : Install patroni] *******************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/var/lib/postgresql/15/main" exists] *************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/var/lib/postgresql/15/main" is empty] ******************************************************************************************************
changed: [10.0.1.1] => (item=absent)
changed: [10.0.1.2] => (item=absent)
changed: [10.0.1.3] => (item=absent)
changed: [10.0.1.1] => (item=directory)
changed: [10.0.1.2] => (item=directory)
changed: [10.0.1.3] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.0.1.1]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.0.1.1]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.0.1.1]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
changed: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.0.1.2]
changed: [10.0.1.3]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [patroni : Disable "postgresql@15-main" service] **********************************************************************************************************************************************************
ok: [10.0.1.3]
ok: [10.0.1.1]
ok: [10.0.1.2]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.0.1.1]
changed: [10.0.1.3]
changed: [10.0.1.2]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.0.1.1] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.2]
ok: [10.0.1.3]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.0.1.1]
ok: [10.0.1.3]
ok: [10.0.1.2]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.0.1.1]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "users_result.stdout_lines": [
        "                                    List of roles",
        " Role name  |                         Attributes                         | Member of ",
        "------------+------------------------------------------------------------+-----------",
        " pgbouncer  |                                                            | {}",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}",
        " replicator | Replication                                                | {}"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7453 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +----------+---------+-----------+----+-----------+",
        "| Member                          | Host     | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+----------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+----------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.0.1.1]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "msg": [
        "+------------------------------------------------+",
        "address 10.0.1.1,10.0.1.2,10.0.1.3",
        "port 6432 (pgbouncer)",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.0.1.1                   : ok=104  changed=58   unreachable=0    failed=0    skipped=348  rescued=0    ignored=0   
10.0.1.2                   : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
10.0.1.3                   : ok=89   changed=55   unreachable=0    failed=0    skipped=328  rescued=0    ignored=0   
localhost                  : ok=24   changed=6    unreachable=0    failed=0    skipped=107  rescued=0    ignored=0

passed

vitabaks commented 7 months ago

Test: Azure

export AZURE_SUBSCRIPTION_ID=********
export AZURE_CLIENT_ID=********
export AZURE_SECRET=********
export AZURE_TENANT=********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='azure' \
     servers_count=3 \
     server_type='Standard_DS1_v2' \
     server_location='eastus' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster"

Result:

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "etcd")] ********************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that Azure collection is installed on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Get ansible_collections path] **********************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that Azure collection requirements is present on controlling host] **************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Azure: Create resource group] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create virtual network] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create subnet] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create public IP address] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-public-ip)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-public-ip)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-public-ip)

TASK [cloud-resources : Azure: Create or modify Security Group] ************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create network interface] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-network-interface)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02-network-interface)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03-network-interface)

TASK [cloud-resources : Azure: Create virtual machine] *********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Show Azure VM info] ********************************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode01",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202309190', 'exactVersion': '22.04.202309190'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 52.188.152.174",
        "Private IP: 10.0.1.4"
    ]
}
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode02) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode02",
        "Name: vitaliy-test-pgcluster-pgnode02",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202309190', 'exactVersion': '22.04.202309190'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 40.114.91.246",
        "Private IP: 10.0.1.5"
    ]
}
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode03) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode03",
        "Name: vitaliy-test-pgcluster-pgnode03",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202309190', 'exactVersion': '22.04.202309190'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 13.92.27.118",
        "Private IP: 10.0.1.6"
    ]
}

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode02)
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode03)

TASK [cloud-resources : Inventory | Initialize ip_addresses variable] ******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Inventory | Extract IP addresses] ******************************************************************************************************************************************************
ok: [localhost] => (item=public_ip: 52.188.152.174, private_ip: 10.0.1.4)
ok: [localhost] => (item=public_ip: 40.114.91.246, private_ip: 10.0.1.5)
ok: [localhost] => (item=public_ip: 13.92.27.118, private_ip: 10.0.1.6)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'master' groups] ***************************************************************************************************************************
ok: [localhost] => (item=52.188.152.174)

TASK [cloud-resources : Inventory | Add host to 'postgres_cluster', 'replica' groups] **************************************************************************************************************************
ok: [localhost] => (item=40.114.91.246)
ok: [localhost] => (item=13.92.27.118)

TASK [cloud-resources : Inventory | Add host to 'etcd_cluster' group] ******************************************************************************************************************************************
ok: [localhost] => (item=52.188.152.174)
ok: [localhost] => (item=40.114.91.246)
ok: [localhost] => (item=13.92.27.118)

PLAY [deploy_pgcluster.yml | Perform pre-checks] ***************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.6]
ok: [10.0.1.4]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [System information] **************************************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz, count: 1, cores: 1",
        "Disk space total": "84.46 GB",
        "Kernel": "6.2.0-1012-azure",
        "Memory": "3.32 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Virtual Machine",
        "Virtualization type": "VirtualPC"
    }
}
ok: [10.0.1.5] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz, count: 1, cores: 1",
        "Disk space total": "84.46 GB",
        "Kernel": "6.2.0-1012-azure",
        "Memory": "3.32 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Virtual Machine",
        "Virtualization type": "VirtualPC"
    }
}
ok: [10.0.1.6] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz, count: 1, cores: 1",
        "Disk space total": "84.46 GB",
        "Kernel": "6.2.0-1012-azure",
        "Memory": "3.32 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Virtual Machine",
        "Virtualization type": "VirtualPC"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***********************************************************************************************************************************************
ok: [10.0.1.4] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ************************************************************************************************************************************************************
ok: [10.0.1.4] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] ******************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pre-checks : PgBouncer | Show total pool_size] ***********************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/pgdata/16/main" is not initialized] ****************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pre-checks : Generate a password for patroni superuser] **************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pre-checks : Generate a password for patroni replication user] *******************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pre-checks : Generate a password for pgbouncer auth user] ************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [Make sure that the iproute is installed] *****************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

PLAY [Configure etcd Cluster and System Settings] **************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [Update apt cache] ****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.4] => (item=etcd_cluster)
ok: [10.0.1.4] => (item=master)
ok: [10.0.1.4] => (item=postgres_cluster)
ok: [10.0.1.5] => (item=etcd_cluster)
ok: [10.0.1.5] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=etcd_cluster)
ok: [10.0.1.5] => (item=replica)
ok: [10.0.1.6] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.5] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.6] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
changed: [10.0.1.4] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.5] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [10.0.1.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.6] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.5] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [10.0.1.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.6] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.5] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
changed: [10.0.1.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.6] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.5] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [10.0.1.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.5] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.6] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [10.0.1.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.6] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.5] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [10.0.1.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.6] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.5] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [10.0.1.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.6] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.5] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
changed: [10.0.1.6] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.5] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.6] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
changed: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
changed: [10.0.1.5] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
changed: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [etcd : Make sure the unzip/tar packages are present] *****************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Download "etcd" package] **************************************************************************************************************************************************************************
changed: [10.0.1.4] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
[WARNING]: Module remote_tmp /tmp/root/ansible did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct
permissions manually
changed: [10.0.1.6] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)
changed: [10.0.1.5] => (item=https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz)

TASK [etcd : Extract "etcd" into /tmp] *************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [etcd : Copy "etcd" and "etcdctl" binary files to /usr/local/bin/] ****************************************************************************************************************************************
changed: [10.0.1.6] => (item=etcd)
changed: [10.0.1.4] => (item=etcd)
changed: [10.0.1.5] => (item=etcd)
changed: [10.0.1.4] => (item=etcdctl)
changed: [10.0.1.6] => (item=etcdctl)
changed: [10.0.1.5] => (item=etcdctl)

TASK [etcd : Add etcd user] ************************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Create etcd conf directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Generate conf file "/etc/etcd/etcd.conf"] *********************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Copy systemd service file] ************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Enable and start etcd service] ********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [etcd : Wait for port 2379 to become open on the host] ****************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [etcd : Wait until the etcd cluster is healthy] ***********************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [etcd : cluster health] ***********************************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": "http://10.0.1.4:2379 is healthy: successfully committed proposal: took = 3.555672ms\n"
}
ok: [10.0.1.5] => {
    "msg": "http://10.0.1.5:2379 is healthy: successfully committed proposal: took = 6.794688ms\n"
}
ok: [10.0.1.6] => {
    "msg": "http://10.0.1.6:2379 is healthy: successfully committed proposal: took = 5.695698ms\n"
}

PLAY [consul.yml | Consul Playbook] ****************************************************************************************************************************************************************************

PLAY [consul.yml | Configure Consul instances] *****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Postgres Cluster Configuration] ***************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [add-repository : Add repository apt-key] *****************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.6] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})
changed: [10.0.1.5] => (item={'key': 'https://www.postgresql.org/media/keys/ACCC4CF8.asc'})

TASK [add-repository : Add repository] *************************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.6] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})
changed: [10.0.1.5] => (item={'repo': 'deb https://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main'})

TASK [packages : Install system packages] **********************************************************************************************************************************************************************
ok: [10.0.1.4] => (item=['python3'])
ok: [10.0.1.5] => (item=['python3'])
ok: [10.0.1.6] => (item=['python3'])
ok: [10.0.1.4] => (item=python3)
ok: [10.0.1.6] => (item=python3)
ok: [10.0.1.5] => (item=python3)
changed: [10.0.1.4] => (item=python3-dev)
changed: [10.0.1.6] => (item=python3-dev)
changed: [10.0.1.4] => (item=python3-psycopg2)
changed: [10.0.1.5] => (item=python3-dev)
ok: [10.0.1.4] => (item=python3-setuptools)
changed: [10.0.1.6] => (item=python3-psycopg2)
ok: [10.0.1.6] => (item=python3-setuptools)
changed: [10.0.1.5] => (item=python3-psycopg2)
ok: [10.0.1.5] => (item=python3-setuptools)
changed: [10.0.1.4] => (item=python3-pip)
ok: [10.0.1.4] => (item=curl)
ok: [10.0.1.4] => (item=less)
ok: [10.0.1.4] => (item=sudo)
ok: [10.0.1.4] => (item=vim)
ok: [10.0.1.4] => (item=gcc)
changed: [10.0.1.6] => (item=python3-pip)
ok: [10.0.1.6] => (item=curl)
changed: [10.0.1.4] => (item=jq)
changed: [10.0.1.5] => (item=python3-pip)
ok: [10.0.1.4] => (item=iptables)
ok: [10.0.1.6] => (item=less)
ok: [10.0.1.5] => (item=curl)
ok: [10.0.1.6] => (item=sudo)
ok: [10.0.1.5] => (item=less)
ok: [10.0.1.6] => (item=vim)
changed: [10.0.1.4] => (item=acl)
ok: [10.0.1.6] => (item=gcc)
ok: [10.0.1.5] => (item=sudo)
ok: [10.0.1.5] => (item=vim)
changed: [10.0.1.4] => (item=dnsutils)
ok: [10.0.1.5] => (item=gcc)
changed: [10.0.1.6] => (item=jq)
ok: [10.0.1.6] => (item=iptables)
changed: [10.0.1.5] => (item=jq)
ok: [10.0.1.5] => (item=iptables)
changed: [10.0.1.6] => (item=acl)
changed: [10.0.1.6] => (item=dnsutils)
changed: [10.0.1.5] => (item=acl)
changed: [10.0.1.5] => (item=dnsutils)

TASK [packages : PostgreSQL | ensure postgresql database-cluster manager package] ******************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [packages : PostgreSQL | disable initializing of a default postgresql cluster] ****************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [packages : PostgreSQL | disable log rotation with logrotate for postgresql] ******************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [packages : Install PostgreSQL packages] ******************************************************************************************************************************************************************
changed: [10.0.1.4] => (item=postgresql-16)
ok: [10.0.1.4] => (item=postgresql-client-16)
ok: [10.0.1.4] => (item=postgresql-contrib-16)
changed: [10.0.1.6] => (item=postgresql-16)
ok: [10.0.1.6] => (item=postgresql-client-16)
changed: [10.0.1.5] => (item=postgresql-16)
ok: [10.0.1.6] => (item=postgresql-contrib-16)
ok: [10.0.1.5] => (item=postgresql-client-16)
ok: [10.0.1.5] => (item=postgresql-contrib-16)
changed: [10.0.1.4] => (item=postgresql-server-dev-16)
changed: [10.0.1.6] => (item=postgresql-server-dev-16)
changed: [10.0.1.5] => (item=postgresql-server-dev-16)

TASK [sudo : Add user to /etc/sudoers.d/] **********************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.5] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})
changed: [10.0.1.6] => (item={'name': 'postgres', 'nopasswd': 'yes', 'commands': 'ALL'})

TASK [swap : Ensure swap exists] *******************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [swap : Create swap file] *********************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [swap : Set permissions on swap file] *********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [swap : Make swap file if necessary] **********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [swap : Run swapon on the swap file] **********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [swap : Manage swap file entry in fstab] ******************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [sysctl : Build a sysctl_conf dynamic variable] ***********************************************************************************************************************************************************
ok: [10.0.1.4] => (item=etcd_cluster)
ok: [10.0.1.4] => (item=master)
ok: [10.0.1.5] => (item=etcd_cluster)
ok: [10.0.1.4] => (item=postgres_cluster)
ok: [10.0.1.5] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=etcd_cluster)
ok: [10.0.1.5] => (item=replica)
ok: [10.0.1.6] => (item=postgres_cluster)
ok: [10.0.1.6] => (item=replica)

TASK [sysctl : Setting kernel parameters] **********************************************************************************************************************************************************************
ok: [10.0.1.4] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.5] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.6] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [10.0.1.4] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.6] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [10.0.1.4] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.5] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.6] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [10.0.1.4] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.5] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.6] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [10.0.1.4] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.6] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.5] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [10.0.1.4] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.6] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.5] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [10.0.1.4] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.6] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.5] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [10.0.1.4] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.6] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.5] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
ok: [10.0.1.4] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.6] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.5] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [10.0.1.4] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.6] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.5] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.4] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [10.0.1.6] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.4] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [10.0.1.6] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [10.0.1.5] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})

TASK [transparent_huge_pages : Create systemd service "disable-transparent-huge-pages.service"] ****************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

RUNNING HANDLER [transparent_huge_pages : Start disable-transparent-huge-pages service] ************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pam_limits : Linux PAM limits | add or modify nofile limits] *********************************************************************************************************************************************
changed: [10.0.1.4] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.5] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.6] => (item={'limit_type': 'soft', 'limit_item': 'nofile', 'value': 65536})
changed: [10.0.1.4] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.5] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})
changed: [10.0.1.6] => (item={'limit_type': 'hard', 'limit_item': 'nofile', 'value': 200000})

TASK [locales : Generate locales] ******************************************************************************************************************************************************************************
ok: [10.0.1.4] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.6] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})
ok: [10.0.1.5] => (item={'language_country': 'en_US', 'encoding': 'UTF-8'})

TASK [locales : Set locale "en_US.utf-8" into /etc/default/locale] *********************************************************************************************************************************************
changed: [10.0.1.4] => (item=LANG=en_US.utf-8)
changed: [10.0.1.6] => (item=LANG=en_US.utf-8)
changed: [10.0.1.5] => (item=LANG=en_US.utf-8)
changed: [10.0.1.4] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.6] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.5] => (item=LANGUAGE=en_US.utf-8)
changed: [10.0.1.4] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.6] => (item=LC_ALL=en_US.utf-8)
changed: [10.0.1.5] => (item=LC_ALL=en_US.utf-8)

TASK [mount : Detect empty volume] *****************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [mount : Create "ext4" filesystem on the disk "/dev/sdc"] *************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.5]
changed: [10.0.1.4]

TASK [mount : Get UUID of the disk "/dev/sdc"] *****************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [mount : Set variable: mount.src] *************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [mount : Mount the filesystem] ****************************************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'src': '56a254c9-efd8-4f2d-ae12-dc55036a747f', 'path': '/pgdata'})
changed: [10.0.1.6] => (item={'src': '2c71f0fa-29d3-4be4-ab4e-342787754147', 'path': '/pgdata'})
changed: [10.0.1.5] => (item={'src': '96cdeacf-9935-4ec8-8674-df82e0477a90', 'path': '/pgdata'})

PLAY [balancers.yml | Configure load balancers] ****************************************************************************************************************************************************************
skipping: no hosts matched

PLAY [deploy_pgcluster.yml | Install and configure pgBackRest] *************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

PLAY [deploy_pgcluster.yml | PostgreSQL Cluster Deployment] ****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [Include OS-specific variables] ***************************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [cron : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer : Install pgbouncer package] *******************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pgbouncer : Ensure config directory "/etc/pgbouncer" exist] **********************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [pgbouncer : Check if pgbouncer systemd service file exists] **********************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer : Stop and disable standard init script] *******************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pgbouncer : Configure pgbouncer systemd service file] ****************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [pgbouncer : Ensure pgbouncer service is enabled] *********************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.5]
changed: [10.0.1.4]

TASK [pgbouncer : Enable log rotation with logrotate] **********************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.4]
changed: [10.0.1.5]

TASK [pgbouncer : Configure pgbouncer.ini] *********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [pgpass : Configure a password file (/var/lib/postgresql/.pgpass)] ****************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

RUNNING HANDLER [pgbouncer : Restart pgbouncer service] ********************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

RUNNING HANDLER [pgbouncer : Wait for port "6432" to become open on the host] **********************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.5]
ok: [10.0.1.6]

RUNNING HANDLER [Restart pgbouncer service] ********************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

RUNNING HANDLER [Wait for port "6432" to become open on the host] **********************************************************************************************************************************************
ok: [10.0.1.6]
ok: [10.0.1.5]
ok: [10.0.1.4]

TASK [patroni : Copy patroni requirements.txt file] ************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Install setuptools] ****************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Install requirements] **************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Install patroni 3.1.0] *************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Create conf directory] *************************************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Generate conf file "/etc/patroni/patroni.yml"] *************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Copy systemd service file "/etc/systemd/system/patroni.service"] *******************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql log directory "/var/log/postgresql" exists] ******************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.5]
changed: [10.0.1.6]

TASK [patroni : Prepare PostgreSQL | make sure PostgreSQL data directory "/pgdata/16/main" exists] *************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | make sure the postgresql config files exists] *****************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | generate default postgresql config files] *********************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | make sure the data directory "/pgdata/16/main" is empty] ******************************************************************************************************************
changed: [10.0.1.4] => (item=absent)
changed: [10.0.1.6] => (item=absent)
changed: [10.0.1.5] => (item=absent)
changed: [10.0.1.4] => (item=directory)
changed: [10.0.1.6] => (item=directory)
changed: [10.0.1.5] => (item=directory)

TASK [patroni : Start patroni service on the Master server] ****************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [patroni : Check PostgreSQL is started and accepting connections on Master] *******************************************************************************************************************************
ok: [10.0.1.4]

TASK [patroni : Wait for the cluster to initialize (master is the leader with the lock)] ***********************************************************************************************************************
ok: [10.0.1.4]

TASK [patroni : Prepare PostgreSQL | generate pg_hba.conf] *****************************************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Prepare PostgreSQL | reload for apply the pg_hba.conf] *****************************************************************************************************************************************
changed: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Start patroni service on Replica servers] ******************************************************************************************************************************************************
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Wait for port 8008 to become open on the host] *************************************************************************************************************************************************
ok: [10.0.1.5]
ok: [10.0.1.6]

TASK [patroni : Check that the patroni is healthy on the replica server] ***************************************************************************************************************************************
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Turning off postgresql autostart from config "start.conf" (will be managed by patroni)] ********************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [patroni : Disable "postgresql@16-main" service] **********************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [patroni : Add PATRONICTL_CONFIG_FILE environment variable into /etc/environment] *************************************************************************************************************************
changed: [10.0.1.4]
changed: [10.0.1.6]
changed: [10.0.1.5]

TASK [postgresql-users : Make sure the PostgreSQL users are present] *******************************************************************************************************************************************
changed: [10.0.1.4] => (item=pgbouncer)

TASK [pgbouncer/config : Ensure config directory "/etc/pgbouncer" exist] ***************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer/config : Update pgbouncer.ini] *****************************************************************************************************************************************************************
ok: [10.0.1.4]
ok: [10.0.1.6]
ok: [10.0.1.5]

TASK [pgbouncer/config : Create function 'user_search' for pgbouncer 'auth_query' option in all databases] *****************************************************************************************************
changed: [10.0.1.4]

TASK [deploy-finish : Get postgresql users list] ***************************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL list of users] ****************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Get postgresql database list] ************************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL list of databases] ************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "dbs_result.stdout_lines": [
        "   name   |  owner   | encoding |   collate   |    ctype    |  size   | tablespace ",
        "----------+----------+----------+-------------+-------------+---------+------------",
        " postgres | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 7492 kB | pg_default",
        "(1 row)"
    ]
}

TASK [deploy-finish : Check postgresql cluster health] *********************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL Cluster health] ***************************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitaliy-test-pgcluster +----------+---------+-----------+----+-----------+",
        "| Member                          | Host     | Role    | State     | TL | Lag in MB |",
        "+---------------------------------+----------+---------+-----------+----+-----------+",
        "| vitaliy-test-pgcluster-pgnode01 | 10.0.1.4 | Leader  | running   |  1 |           |",
        "| vitaliy-test-pgcluster-pgnode02 | 10.0.1.5 | Replica | streaming |  1 |         0 |",
        "| vitaliy-test-pgcluster-pgnode03 | 10.0.1.6 | Replica | streaming |  1 |         0 |",
        "+---------------------------------+----------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Create list of nodes] ********************************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [deploy-finish : PostgreSQL Cluster connection info] ******************************************************************************************************************************************************
ok: [10.0.1.4] => {
    "msg": [
        "+------------------------------------------------+",
        "address: 10.0.1.4,10.0.1.5,10.0.1.6",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: Yehap75g3TIZbjc7vjVgJBvaLiWckRFz",
        "+------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.0.1.4                   : ok=112  changed=60   unreachable=0    failed=0    skipped=355  rescued=0    ignored=0   
10.0.1.5                   : ok=94   changed=57   unreachable=0    failed=0    skipped=333  rescued=0    ignored=0   
10.0.1.6                   : ok=94   changed=57   unreachable=0    failed=0    skipped=333  rescued=0    ignored=0   
localhost                  : ok=18   changed=4    unreachable=0    failed=0    skipped=153  rescued=0    ignored=0

passed

vitabaks commented 5 months ago

Test: Deploying to AWS with backups (PgBackRest)

export AWS_ACCESS_KEY_ID=******
export AWS_SECRET_ACCESS_KEY=************

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision=aws \
     servers_count=1 \
     server_type=t3.large \
     server_image=ami-07d445a5585fce0a1 \
     server_location=us-east-2 \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true \
     PGBACKREST_S3_KEY=${AWS_ACCESS_KEY_ID} \
     PGBACKREST_S3_KEY_SECRET=${AWS_SECRET_ACCESS_KEY}"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] *******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Remove temporary SSH key 'ssh_key_tmp_rqsgybf' from cloud (if any)] ****************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Add SSH key 'ssh_key_tmp_rqsgybf' to cloud] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Gather information about default VPC] **********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] ***************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create S3 bucket 'backups-vitaliy-test-pgcluster'] *********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] ******************************************************************************************************************************************
ok: [localhost] => (item=18.226.49.185)

TASK [cloud-resources : Show EC2 instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=18.226.49.185) => {
    "msg": [
        "ID: i-0b233bcdfe89d78fc",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: ami-07d445a5585fce0a1",
        "Type: t3.large",
        "Volume Size: 100 GB",
        "Public IP: 18.226.49.185",
        "Private IP: 172.31.5.233"
    ]
}
...

TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in the AWS S3 bucket] ******************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Make sure pgdg apt key is installed] *********************************************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Make sure pgdg repository is installed] ******************************************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Update apt cache] ****************************************************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Install pgbackrest] **************************************************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Ensure spool directory exist] ****************************************************************************************************************************************************************
ok: [172.31.5.233] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] ***************************************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] ******************************************************************************************************************************************
changed: [172.31.5.233]

TASK [pgbackrest : Make sure that the cron package is installed] ************************************************************************************************************************************************
ok: [172.31.5.233]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] *************************************************************************************************************************************************
changed: [172.31.5.233] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [172.31.5.233] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ****************************************************************************************************************************************
changed: [172.31.5.233]
...

ubuntu@ip-172-31-5-233:~$ cat /etc/pgbackrest/pgbackrest.conf
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=s3
repo1-path=/pgbackrest
repo1-s3-key=******
repo1-s3-key-secret=************
repo1-s3-bucket=backups-vitaliy-test-pgcluster
repo1-s3-endpoint=s3.us-east-2.amazonaws.com
repo1-s3-region=us-east-2
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

ubuntu@ip-172-31-5-233:~$ sudo psql -U postgres
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select * from pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786DB8
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-26 19:25:21.590373+00 |            0 |                 |                  | 2023-11-26 19:22:52.861852+00
(1 row)

postgres=# \q

ubuntu@ip-172-31-5-233:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup

ubuntu@ip-172-31-5-233:~$ sudo su - postgres
postgres@ip-172-31-5-233:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-11-26 19:26:04.158 P00   INFO: backup command begin 2.48: --exec-id=22857-942e136d --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-block --repo1-bundle --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=s3.us-east-2.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-2 --repo1-type=s3 --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-11-26 19:26:04.924 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-11-26 19:26:05.625 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000028
2023-11-26 19:26:05.625 P00   INFO: check archive for prior segment 000000010000000000000002
2023-11-26 19:26:07.280 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-11-26 19:26:07.480 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000138
2023-11-26 19:26:07.522 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-11-26 19:26:07.860 P00   INFO: new backup label = 20231126-192604F
2023-11-26 19:26:08.275 P00   INFO: full backup size = 22.3MB, file total = 972
2023-11-26 19:26:08.275 P00   INFO: backup command end: completed successfully (4121ms)
2023-11-26 19:26:08.276 P00   INFO: expire command begin 2.48: --exec-id=22857-942e136d --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=s3.us-east-2.amazonaws.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=us-east-2 --repo1-type=s3 --stanza=vitaliy-test-pgcluster
2023-11-26 19:26:08.362 P00   INFO: expire command end: completed successfully (87ms)
postgres@ip-172-31-5-233:~$ 
postgres@ip-172-31-5-233:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231126-192604F
            timestamp start/stop: 2023-11-26 19:26:04+00 / 2023-11-26 19:26:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB
postgres@ip-172-31-5-233:~$

passed

vitabaks commented 5 months ago

Test: Deploying to AWS with backups (WAL-G)

export AWS_ACCESS_KEY_ID=******
export AWS_SECRET_ACCESS_KEY=************

ansible-playbook deploy_pgcluster.yml \
  --user=ubuntu --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision=aws \
     servers_count=1 \
     server_type=t3.large \
     server_image=ami-07d445a5585fce0a1 \
     server_location=us-east-2 \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-walg-pgcluster \
     wal_g_install=true \
     WALG_AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
     WALG_AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] *******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Remove temporary SSH key 'ssh_key_tmp_xllykek' from cloud (if any)] ****************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Add SSH key 'ssh_key_tmp_xllykek' to cloud] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Gather information about default VPC] **********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] ***************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create S3 bucket 'backups-vitaliy-test-walg-pgcluster'] ****************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-walg-pgcluster-pgnode01)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] ******************************************************************************************************************************************
ok: [localhost] => (item=3.15.139.143)

TASK [cloud-resources : Show EC2 instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=3.15.139.143) => {
    "msg": [
        "ID: i-0275355ef89391ed9",
        "Name: vitaliy-test-walg-pgcluster-pgnode01",
        "Image: ami-07d445a5585fce0a1",
        "Type: t3.large",
        "Volume Size: 100 GB",
        "Public IP: 3.15.139.143",
        "Private IP: 172.31.1.233"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in the AWS S3 bucket] ****************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Check if WAL-G is already installed] **************************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Install lib dependencies to build WAL-G] **********************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Check the installed Go version] *******************************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Check the latest available Go version] ************************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Download Go v1.21.4] ******************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Install Go] ***************************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Download WAL-G v2.0.1 source code] ****************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Build WAL-G deps] *********************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Build and install WAL-G] **************************************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ************************************************************************************************************************************************
changed: [172.31.1.233]

TASK [wal-g : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [172.31.1.233]

TASK [wal-g : Add WAL-G cron jobs] ******************************************************************************************************************************************************************************
changed: [172.31.1.233] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test-walg-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [172.31.1.233] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

ubuntu@ip-172-31-1-233:~$ cat /var/lib/postgresql/.walg.json
{
  "AWS_ACCESS_KEY_ID": "******",
  "AWS_SECRET_ACCESS_KEY": "*********",
  "WALG_S3_PREFIX": "s3://backups-vitaliy-test-walg-pgcluster",
  "AWS_REGION": "us-east-2",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test-walg-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}

ubuntu@ip-172-31-1-233:~$ sudo psql -U postgres
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D40
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-26 19:44:50.612286+00 |            0 |                 |                  | 2023-11-26 19:34:09.001261+00
(1 row)

postgres=# \q
ubuntu@ip-172-31-1-233:~$ sudo su - postgres
postgres@ip-172-31-1-233:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test-walg-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://172.31.1.233:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@ip-172-31-1-233:~$ 
postgres@ip-172-31-1-233:~$ wal-g backup-push /pgdata/16/vitaliy-test-walg-pgcluster
INFO: 2023/11/26 19:45:22.202237 Couldn't find previous backup. Doing full backup.
INFO: 2023/11/26 19:45:22.212877 Calling pg_start_backup()
INFO: 2023/11/26 19:45:22.383367 Starting a new tar bundle
INFO: 2023/11/26 19:45:22.383416 Walking ...
INFO: 2023/11/26 19:45:22.384018 Starting part 1 ...
INFO: 2023/11/26 19:45:22.713382 Packing ...
INFO: 2023/11/26 19:45:22.714229 Finished writing part 1.
INFO: 2023/11/26 19:45:22.850398 Starting part 2 ...
INFO: 2023/11/26 19:45:22.851227 /global/pg_control
INFO: 2023/11/26 19:45:22.854428 Finished writing part 2.
INFO: 2023/11/26 19:45:22.854458 Calling pg_stop_backup()
INFO: 2023/11/26 19:45:22.908507 Starting part 3 ...
INFO: 2023/11/26 19:45:22.908578 backup_label
INFO: 2023/11/26 19:45:22.908591 tablespace_map
INFO: 2023/11/26 19:45:22.929796 Finished writing part 3.
INFO: 2023/11/26 19:45:23.121834 Wrote backup with name base_000000010000000000000003
postgres@ip-172-31-1-233:~$ 
postgres@ip-172-31-1-233:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-11-26T19:45:24Z 000000010000000000000003
postgres@ip-172-31-1-233:~$

passed

vitabaks commented 5 months ago

Test: Deploying to GCP with backups (PgBackRest)

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  *********
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='gcp' \
     servers_count=1 \
     server_type='n2-standard-2' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     server_location='us-central1' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'google-auth' dependency is present on controlling host] *******************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : GCP: Gather information about project] *************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_network_name] ********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Gather information about network] *************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Extract ip_range for network 'default'] *******************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Create or modify SSH public firewall] *********************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify Postgres cluster firewall] ***************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create bucket 'backups-vitaliy-test-pgcluster'] ***********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify VM instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=35.239.57.82)

TASK [cloud-resources : Show GCP instance info] ****************************************************************************************************************************************************************
ok: [localhost] => (item=35.239.57.82) => {
    "msg": [
        "ID: 1115300626491322267",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: ubuntu-2204-lts",
        "Type: n2-standard-2",
        "Volume Size: 100 GB",
        "Public IP: 35.239.57.82",
        "Private IP: 10.128.0.2"
    ]
}
...
TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in the GCS Bucket] ********************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Get GCP service account contents from localhost] ********************************************************************************************************************************************
ok: [10.128.0.2 -> localhost]

TASK [pgbackrest : Copy GCP service account contents to /var/lib/postgresql/gcs-key.json] **********************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ***************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Make sure pgdg apt key is installed] ********************************************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Make sure pgdg repository is installed] *****************************************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Update apt cache] ***************************************************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Install pgbackrest] *************************************************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Ensure spool directory exist] ***************************************************************************************************************************************************************
ok: [10.128.0.2] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] **************************************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] *****************************************************************************************************************************************
changed: [10.128.0.2]

TASK [pgbackrest : Make sure that the cron package is installed] ***********************************************************************************************************************************************
ok: [10.128.0.2]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] ************************************************************************************************************************************************
changed: [10.128.0.2] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [10.128.0.2] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ***************************************************************************************************************************************
changed: [10.128.0.2]
...

postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/pgbackrest/pgbackrest.conf 
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=gcs
repo1-path=/pgbackrest
repo1-gcs-key=/var/lib/postgresql/gcs-key.json
repo1-gcs-bucket=backups-vitaliy-test-pgcluster
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

postgres@vitaliy-test-pgcluster-pgnode01:~$ ls -l /var/lib/postgresql/gcs-key.json
-rw------- 1 postgres postgres 2309 Nov 27 13:01 /var/lib/postgresql/gcs-key.json
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: error (no valid backups)
    cipher: none

    db (current)
        wal archive min/max (16): none present
postgres@vitaliy-test-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D40
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-27 13:05:38.828697+00 |            0 |                 |                  | 2023-11-27 13:03:29.574019+00
(1 row)

postgres=# \q
postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-11-27 13:06:03.308 P00   INFO: backup command begin 2.48: --exec-id=16256-0a838f5f --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-block --repo1-bundle --repo1-gcs-bucket=backups-vitaliy-test-pgcluster --repo1-gcs-key=<redacted> --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=gcs --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-11-27 13:06:04.206 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-11-27 13:06:04.907 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000028
2023-11-27 13:06:04.907 P00   INFO: check archive for prior segment 000000010000000000000002
2023-11-27 13:06:07.633 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-11-27 13:06:07.833 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000100
2023-11-27 13:06:07.991 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-11-27 13:06:08.375 P00   INFO: new backup label = 20231127-130604F
2023-11-27 13:06:09.590 P00   INFO: full backup size = 22.3MB, file total = 972
2023-11-27 13:06:09.591 P00   INFO: backup command end: completed successfully (6285ms)
2023-11-27 13:06:09.591 P00   INFO: expire command begin 2.48: --exec-id=16256-0a838f5f --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-gcs-bucket=backups-vitaliy-test-pgcluster --repo1-gcs-key=<redacted> --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=gcs --stanza=vitaliy-test-pgcluster
2023-11-27 13:06:10.093 P00   INFO: expire command end: completed successfully (502ms)
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231127-130604F
            timestamp start/stop: 2023-11-27 13:06:04+00 / 2023-11-27 13:06:07+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB
postgres@vitaliy-test-pgcluster-pgnode01:~$

passed

vitabaks commented 5 months ago

Test: Deploying to GCP with backups (WAL-G)

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  *********
}'

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='gcp' \
     servers_count=1 \
     server_type='n2-standard-2' \
     server_image='projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts' \
     server_location='us-central1' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test2-pgcluster \
     wal_g_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'google-auth' dependency is present on controlling host] ********************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : GCP: Gather information about project] **************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: gcp_network_name] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Gather information about network] **************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Extract ip_range for network 'default'] ********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : GCP: Create or modify SSH public firewall] **********************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify Postgres cluster firewall] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create bucket 'backups-vitaliy-test2-pgcluster'] ***********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : GCP: Create or modify VM instance] ******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=35.226.211.79)

TASK [cloud-resources : Show GCP instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=35.226.211.79) => {
    "msg": [
        "ID: 2881656607585414311",
        "Name: vitaliy-test2-pgcluster-pgnode01",
        "Image: ubuntu-2204-lts",
        "Type: n2-standard-2",
        "Volume Size: 100 GB",
        "Public IP: 35.226.211.79",
        "Private IP: 10.128.0.3"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in the AWS S3 bucket] ****************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Get GCP service account contents from localhost] **************************************************************************************************************************************************
ok: [10.128.0.3 -> localhost]

TASK [wal-g : Copy GCP service account contents to /var/lib/postgresql/gcs-key.json] ****************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Check if WAL-G is already installed] **************************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Install lib dependencies to build WAL-G] **********************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Check the installed Go version] *******************************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Check the latest available Go version] ************************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Download Go v1.21.4] ******************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Install Go] ***************************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Download WAL-G v2.0.1 source code] ****************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Build WAL-G deps] *********************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Build and install WAL-G] **************************************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ************************************************************************************************************************************************
changed: [10.128.0.3]

TASK [wal-g : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.128.0.3]

TASK [wal-g : Add WAL-G cron jobs] ******************************************************************************************************************************************************************************
changed: [10.128.0.3] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [10.128.0.3] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

postgres@vitaliy-test2-pgcluster-pgnode01:~$ ls -la
total 20
drwxr-xr-x  2 postgres postgres 4096 Nov 27 13:11 .
drwxr-xr-x 41 root     root     4096 Nov 27 13:05 ..
-rw-------  1 postgres postgres  170 Nov 27 13:10 .pgpass
-rw-r--r--  1 postgres postgres  443 Nov 27 13:10 .walg.json
-rw-------  1 postgres postgres 2309 Nov 27 13:05 gcs-key.json
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat .walg.json 
{
  "GOOGLE_APPLICATION_CREDENTIALS": "/var/lib/postgresql/gcs-key.json",
  "WALG_GS_PREFIX": "gs://backups-vitaliy-test2-pgcluster",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test2-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}
postgres@vitaliy-test2-pgcluster-pgnode01:~$ ls -l /var/lib/postgresql/gcs-key.json
-rw------- 1 postgres postgres 2309 Nov 27 13:05 /var/lib/postgresql/gcs-key.json
postgres@vitaliy-test2-pgcluster-pgnode01:~$ less /var/lib/postgresql/gcs-key.json
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-27 13:22:43.167254+00 |            0 |                 |                  | 2023-11-27 13:11:39.862766+00
(1 row)

postgres=# \q
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.128.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster
INFO: 2023/11/27 13:23:10.614574 Couldn't find previous backup. Doing full backup.
INFO: 2023/11/27 13:23:10.624595 Calling pg_start_backup()
INFO: 2023/11/27 13:23:10.820279 Starting a new tar bundle
INFO: 2023/11/27 13:23:10.820321 Walking ...
INFO: 2023/11/27 13:23:10.820533 Starting part 1 ...
INFO: 2023/11/27 13:23:11.073171 Packing ...
INFO: 2023/11/27 13:23:11.073925 Finished writing part 1.
INFO: 2023/11/27 13:23:11.475888 Starting part 2 ...
INFO: 2023/11/27 13:23:11.476047 /global/pg_control
INFO: 2023/11/27 13:23:11.477486 Finished writing part 2.
INFO: 2023/11/27 13:23:11.477513 Calling pg_stop_backup()
INFO: 2023/11/27 13:23:11.527518 Starting part 3 ...
INFO: 2023/11/27 13:23:11.527613 backup_label
INFO: 2023/11/27 13:23:11.527624 tablespace_map
INFO: 2023/11/27 13:23:11.569783 Finished writing part 3.
INFO: 2023/11/27 13:23:13.046710 Wrote backup with name base_000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-11-27T13:23:12Z 000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$

passed

vitabaks commented 5 months ago

Test: Deploying to Azure with backups (PgBackRest)

export AZURE_SUBSCRIPTION_ID=*********
export AZURE_CLIENT_ID=*********
export AZURE_SECRET=*********
export AZURE_TENANT=*********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='azure' \
     servers_count=1 \
     server_type='Standard_DS1_v2' \
     server_location='eastus' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that Azure collection is installed on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Get ansible_collections path] **********************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that Azure collection requirements is present on controlling host] **************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Azure: Create resource group] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create virtual network] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create subnet] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create public IP address] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-public-ip)

TASK [cloud-resources : Azure: Create or modify Security Group] ************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create network interface] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-network-interface)

TASK [cloud-resources : Azure: Create Storage Account 'vitaliytestpgcluster'] **********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Get Storage Account info] *******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: azure_storage_account_key] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create Blob Storage 'backups-vitaliy-test-pgcluster'] ***************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create virtual machine] *********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Show Azure VM info] ********************************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test-pgcluster-pgnode01",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202311010', 'exactVersion': '22.04.202311010'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 172.190.155.240",
        "Private IP: 10.0.1.4"
    ]
}
...
TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in Azure Blob Storage] ****************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ***************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Make sure pgdg apt key is installed] ********************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Make sure pgdg repository is installed] *****************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Update apt cache] ***************************************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Install pgbackrest] *************************************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Ensure spool directory exist] ***************************************************************************************************************************************************************
ok: [10.0.1.4] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] **************************************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] *****************************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest : Make sure that the cron package is installed] ***********************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] ************************************************************************************************************************************************
changed: [10.0.1.4] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [10.0.1.4] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Get repo1-path value] *********************************************************************************************************************************************************
ok: [10.0.1.4]

TASK [pgbackrest/stanza-create : Make sure the /pgbackrest directory exists] ***********************************************************************************************************************************
changed: [10.0.1.4]

TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ***************************************************************************************************************************************
changed: [10.0.1.4]
...

postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/pgbackrest/pgbackrest.conf 
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=azure
repo1-path=/pgbackrest
repo1-azure-key=************
repo1-azure-key-type=shared
repo1-azure-account=vitaliytestpgcluster
repo1-azure-container=backups-vitaliy-test-pgcluster
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

postgres@vitaliy-test-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver ;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-28 14:46:04.139787+00 |            0 |                 |                  | 2023-11-28 14:43:30.124269+00
(1 row)

postgres=# \q
postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-11-28 14:46:26.318 P00   INFO: backup command begin 2.49: --exec-id=11975-627d9fba --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-azure-account=<redacted> --repo1-azure-container=backups-vitaliy-test-pgcluster --repo1-azure-key=<redacted> --repo1-azure-key-type=shared --repo1-block --repo1-bundle --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=azure --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-11-28 14:46:27.058 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-11-28 14:46:27.759 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000060
2023-11-28 14:46:27.763 P00   INFO: check archive for prior segment 000000010000000000000002
2023-11-28 14:46:29.296 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-11-28 14:46:29.497 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000138
2023-11-28 14:46:29.526 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-11-28 14:46:29.840 P00   INFO: new backup label = 20231128-144626F
2023-11-28 14:46:29.990 P00   INFO: full backup size = 22.3MB, file total = 972
2023-11-28 14:46:29.990 P00   INFO: backup command end: completed successfully (3675ms)
2023-11-28 14:46:29.990 P00   INFO: expire command begin 2.49: --exec-id=11975-627d9fba --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-azure-account=<redacted> --repo1-azure-container=backups-vitaliy-test-pgcluster --repo1-azure-key=<redacted> --repo1-azure-key-type=shared --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-type=azure --stanza=vitaliy-test-pgcluster
2023-11-28 14:46:30.032 P00   INFO: expire command end: completed successfully (42ms)
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231128-144626F
            timestamp start/stop: 2023-11-28 14:46:26+00 / 2023-11-28 14:46:29+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB

passed

vitabaks commented 5 months ago

Test: Deploying to Azure with backups (WAL-G)

export AZURE_SUBSCRIPTION_ID=*********
export AZURE_CLIENT_ID=*********
export AZURE_SECRET=*********
export AZURE_TENANT=*********

ansible-playbook deploy_pgcluster.yml \
  --user=azureadmin --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='azure' \
     servers_count=1 \
     server_type='Standard_DS1_v2' \
     server_location='eastus' \
     volume_size=100 \
     ssh_key_content=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
     database_public_access=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test2-pgcluster \
     wal_g_install=true"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that Azure collection is installed on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Get ansible_collections path] **********************************************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that Azure collection requirements is present on controlling host] **************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Azure: Create resource group] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create virtual network] *********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create subnet] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create public IP address] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-public-ip)

TASK [cloud-resources : Azure: Create or modify Security Group] ************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create network interface] *******************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-network-interface)

TASK [cloud-resources : Azure: Create Storage Account 'vitaliytest2pgcluster'] *********************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Get Storage Account info] *******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: azure_storage_account_key] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Azure: Create Blob Storage 'backups-vitaliy-test2-pgcluster'] **************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Azure: Create virtual machine] *********************************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01)

TASK [cloud-resources : Show Azure VM info] ********************************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01) => {
    "msg": [
        "ID: /subscriptions/4b9c12ed-3ec0-46ab-948c-ac9af4b8530d/resourceGroups/postgres-cluster-resource-group/providers/Microsoft.Compute/virtualMachines/vitaliy-test2-pgcluster-pgnode01",
        "Name: vitaliy-test2-pgcluster-pgnode01",
        "Image: {'publisher': 'Canonical', 'offer': '0001-com-ubuntu-server-jammy', 'sku': '22_04-lts-gen2', 'version': '22.04.202311010', 'exactVersion': '22.04.202311010'}",
        "Type: Standard_DS1_v2",
        "Volume Size: 100 GB",
        "Volume Type: StandardSSD_LRS",
        "Public IP: 20.120.18.203",
        "Private IP: 10.0.1.5"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in Azure Blob Storage] **************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Check if WAL-G is already installed] *************************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Install lib dependencies to build WAL-G] *********************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Check the installed Go version] ******************************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Check the latest available Go version] ***********************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Download Go v1.21.4] *****************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Install Go] **************************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Download WAL-G v2.0.1 source code] ***************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Build WAL-G deps] ********************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Build and install WAL-G] *************************************************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ***********************************************************************************************************************************************
changed: [10.0.1.5]

TASK [wal-g : Make sure that the cron package is installed] ****************************************************************************************************************************************************
ok: [10.0.1.5]

TASK [wal-g : Add WAL-G cron jobs] *****************************************************************************************************************************************************************************
changed: [10.0.1.5] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [10.0.1.5] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat .walg.json 
{
  "AZURE_STORAGE_ACCOUNT": "vitaliytest2pgcluster",
  "AZURE_STORAGE_ACCESS_KEY": "************",
  "WALG_AZ_PREFIX": "azure://backups-vitaliy-test2-pgcluster",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test2-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}
postgres@vitaliy-test2-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver();
ERROR:  function pg_stat_archiver() does not exist
LINE 1: select * from pg_stat_archiver();
                      ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
postgres=# select * from pg_stat_archiver ;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-11-28 14:24:42.322069+00 |            0 |                 |                  | 2023-11-28 14:19:28.610455+00
(1 row)

postgres=# \q
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.0.1.5:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster
INFO: 2023/11/28 14:25:14.086633 Couldn't find previous backup. Doing full backup.
INFO: 2023/11/28 14:25:14.098848 Calling pg_start_backup()
INFO: 2023/11/28 14:25:14.427354 Starting a new tar bundle
INFO: 2023/11/28 14:25:14.427384 Walking ...
INFO: 2023/11/28 14:25:14.427561 Starting part 1 ...
INFO: 2023/11/28 14:25:14.682199 Packing ...
INFO: 2023/11/28 14:25:14.683213 Finished writing part 1.
INFO: 2023/11/28 14:25:14.851083 Starting part 2 ...
INFO: 2023/11/28 14:25:14.851536 /global/pg_control
INFO: 2023/11/28 14:25:14.867399 Finished writing part 2.
INFO: 2023/11/28 14:25:14.867600 Calling pg_stop_backup()
INFO: 2023/11/28 14:25:14.967937 Starting part 3 ...
INFO: 2023/11/28 14:25:14.968095 backup_label
INFO: 2023/11/28 14:25:14.968112 tablespace_map
INFO: 2023/11/28 14:25:15.008414 Finished writing part 3.
INFO: 2023/11/28 14:25:15.167885 Wrote backup with name base_000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-11-28T14:25:15Z 000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$

passed

vitabaks commented 5 months ago

Test: Deploying to DigitalOcean with backups (PgBackRest)

export DO_API_TOKEN=*********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='digitalocean' \
     servers_count=1 \
     server_type='g-2vcpu-8gb' \
     server_image='ubuntu-22-04-x64' \
     server_location='nyc1' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test-pgcluster \
     pgbackrest_install=true \
     digital_ocean_spaces_region=nyc3 \
     AWS_ACCESS_KEY_ID='******' \
     AWS_SECRET_ACCESS_KEY='*********'"

Note: Generate an access key and a secret key to access the Spaces API. If you don’t already have keys, you can generate by visiting the control panel’s API page - https://cloud.digitalocean.com/account/api/spaces

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] ***************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get fingerprint for SSH key 'vitaliy'] ************************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] ******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitaliy-test-pgcluster'] ************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify public firewall] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Postgres cluster firewall] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] *************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test-pgcluster-pgnode01)

TASK [cloud-resources : Set variable: droplet_result] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create Spaces Bucket 'backups-vitaliy-test-pgcluster'] ********************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] *******************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] ******************************************************************************************************************************************
changed: [localhost] => (item=vitaliy-test-pgcluster-pgnode01-storage)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=157.230.8.137)

TASK [cloud-resources : Show Droplet info] **********************************************************************************************************************************************************************
ok: [localhost] => (item=157.230.8.137) => {
    "msg": [
        "ID: 387849512",
        "Name: vitaliy-test-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 157.230.8.137",
        "Private IP: 10.116.0.2"
    ]
}
...
TASK [pgbackrest : Set variable 'pgbackrest_conf' for backup in DigitalOcean Spaces Object Storage] *************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Make sure the gnupg and apt-transport-https packages are present] ****************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Make sure pgdg apt key is installed] *********************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Make sure pgdg repository is installed] ******************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Update apt cache] ****************************************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Install pgbackrest] **************************************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Ensure spool directory exist] ****************************************************************************************************************************************************************
ok: [10.116.0.2] => (item=/var/spool/pgbackrest)

TASK [pgbackrest : Ensure config directory exist] ***************************************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Generate conf file /etc/pgbackrest/pgbackrest.conf] ******************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest : Make sure that the cron package is installed] ************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest : Add pgbackrest cron jobs on database server] *************************************************************************************************************************************************
changed: [10.116.0.2] => (item={'name': 'pgBackRest: Full Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '0', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup'})
changed: [10.116.0.2] => (item={'name': 'pgBackRest: Diff Backup', 'file': '/etc/cron.d/pgbackrest-vitaliy-test-pgcluster', 'user': 'postgres', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '1-6', 'job': 'pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup'})
...
TASK [pgbackrest/stanza-create : Get repo1-path value] **********************************************************************************************************************************************************
ok: [10.116.0.2]

TASK [pgbackrest/stanza-create : Make sure the /pgbackrest directory exists] ************************************************************************************************************************************
changed: [10.116.0.2]

TASK [pgbackrest/stanza-create : Create stanza "vitaliy-test-pgcluster"] ****************************************************************************************************************************************
changed: [10.116.0.2]
...

root@vitaliy-test-pgcluster-pgnode01:~# cat /etc/pgbackrest/pgbackrest.conf
[global]
log-level-file=detail
log-path=/var/log/pgbackrest
repo1-type=s3
repo1-path=/pgbackrest
repo1-s3-key=******
repo1-s3-key-secret=*********
repo1-s3-bucket=backups-vitaliy-test-pgcluster
repo1-s3-endpoint=https://nyc3.digitaloceanspaces.com
repo1-s3-region=nyc3
repo1-s3-uri-style=path
repo1-retention-full=4
repo1-retention-archive=4
repo1-retention-archive-type=full
repo1-bundle=y
repo1-block=y
start-fast=y
stop-auto=y
link-all=y
resume=n
archive-async=y
archive-get-queue-max=1GiB
spool-path=/var/spool/pgbackrest
process-max=1

[vitaliy-test-pgcluster]
log-level-console=info
recovery-option=recovery_target_action=promote
pg1-path=/pgdata/16/vitaliy-test-pgcluster

root@vitaliy-test-pgcluster-pgnode01:~# su - postgres
postgres@vitaliy-test-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
                      archive_command                       
------------------------------------------------------------
 pgbackrest --stanza=vitaliy-test-pgcluster archive-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver ;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |         stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+------------------------------
              1 | 000000010000000000000001 | 2023-12-01 11:41:01.950786+00 |            0 |                 |                  | 2023-12-01 11:39:28.53462+00
(1 row)

postgres=# \q
postgres@vitaliy-test-pgcluster-pgnode01:~$ cat /etc/cron.d/pgbackrest-vitaliy-test-pgcluster 
#Ansible: pgBackRest: Full Backup
00 3 * * 0 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
#Ansible: pgBackRest: Diff Backup
00 3 * * 1-6 postgres pgbackrest --stanza=vitaliy-test-pgcluster --type=diff backup
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest --stanza=vitaliy-test-pgcluster --type=full backup
2023-12-01 11:41:38.993 P00   INFO: backup command begin 2.49: --exec-id=24499-ace39a90 --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --pg1-path=/pgdata/16/vitaliy-test-pgcluster --process-max=1 --repo1-block --repo1-bundle --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=https://nyc3.digitaloceanspaces.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=nyc3 --repo1-s3-uri-style=path --repo1-type=s3 --no-resume --stanza=vitaliy-test-pgcluster --start-fast --stop-auto --type=full
2023-12-01 11:41:39.538 P00   INFO: execute non-exclusive backup start: backup begins after the requested immediate checkpoint completes
2023-12-01 11:41:40.239 P00   INFO: backup start archive = 000000010000000000000003, lsn = 0/3000028
2023-12-01 11:41:40.239 P00   INFO: check archive for prior segment 000000010000000000000002
2023-12-01 11:41:42.018 P00   INFO: execute non-exclusive backup stop and wait for all WAL segments to archive
2023-12-01 11:41:42.220 P00   INFO: backup stop archive = 000000010000000000000003, lsn = 0/3000100
2023-12-01 11:41:42.244 P00   INFO: check archive for segment(s) 000000010000000000000003:000000010000000000000003
2023-12-01 11:41:42.367 P00   INFO: new backup label = 20231201-114139F
2023-12-01 11:41:42.562 P00   INFO: full backup size = 22.3MB, file total = 972
2023-12-01 11:41:42.562 P00   INFO: backup command end: completed successfully (3571ms)
2023-12-01 11:41:42.562 P00   INFO: expire command begin 2.49: --exec-id=24499-ace39a90 --log-level-console=info --log-level-file=detail --log-path=/var/log/pgbackrest --repo1-path=/pgbackrest --repo1-retention-archive=4 --repo1-retention-archive-type=full --repo1-retention-full=4 --repo1-s3-bucket=backups-vitaliy-test-pgcluster --repo1-s3-endpoint=https://nyc3.digitaloceanspaces.com --repo1-s3-key=<redacted> --repo1-s3-key-secret=<redacted> --repo1-s3-region=nyc3 --repo1-s3-uri-style=path --repo1-type=s3 --stanza=vitaliy-test-pgcluster
2023-12-01 11:41:42.636 P00   INFO: expire command end: completed successfully (74ms)
postgres@vitaliy-test-pgcluster-pgnode01:~$ 
postgres@vitaliy-test-pgcluster-pgnode01:~$ pgbackrest info
stanza: vitaliy-test-pgcluster
    status: ok
    cipher: none

    db (current)
        wal archive min/max (16): 000000010000000000000001/000000010000000000000003

        full backup: 20231201-114139F
            timestamp start/stop: 2023-12-01 11:41:39+00 / 2023-12-01 11:41:42+00
            wal start/stop: 000000010000000000000003 / 000000010000000000000003
            database size: 22.3MB, database backup size: 22.3MB
            repo1: backup size: 3MB
postgres@vitaliy-test-pgcluster-pgnode01:~$

passed

vitabaks commented 5 months ago

Test: Deploying to DigitalOcean with backups (WAL-G)

export DO_API_TOKEN=*********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "provision='digitalocean' \
     servers_count=1 \
     server_type='g-2vcpu-8gb' \
     server_image='ubuntu-22-04-x64' \
     server_location='nyc1' \
     volume_size=100 \
     ssh_key_name=vitaliy \
     database_public_access=false \
     with_haproxy_load_balancing=false \
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitaliy-test2-pgcluster \
     wal_g_install=true \
     digital_ocean_spaces_region=nyc3 \
     AWS_ACCESS_KEY_ID='*******' \
     AWS_SECRET_ACCESS_KEY='*********'"

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] ***************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get fingerprint for SSH key 'vitaliy'] ************************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] ******************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitaliy-test2-pgcluster'] ***********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify public firewall] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Postgres cluster firewall] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] *************************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01)

TASK [cloud-resources : Set variable: droplet_result] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create Spaces Bucket 'backups-vitaliy-test2-pgcluster'] *******************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] *******************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] ******************************************************************************************************************************************
ok: [localhost] => (item=vitaliy-test2-pgcluster-pgnode01-storage)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=67.205.134.114)

TASK [cloud-resources : Show Droplet info] **********************************************************************************************************************************************************************
ok: [localhost] => (item=67.205.134.114) => {
    "msg": [
        "ID: 387857092",
        "Name: vitaliy-test2-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 67.205.134.114",
        "Private IP: 10.116.0.3"
    ]
}
...
TASK [wal-g : Set variable 'wal_g_json' for backup in DigitalOcean Spaces Object Storage] ***********************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Check if WAL-G is already installed] **************************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Install lib dependencies to build WAL-G] **********************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Check the installed Go version] *******************************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Check the latest available Go version] ************************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Download Go v1.21.4] ******************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Install Go] ***************************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Download WAL-G v2.0.1 source code] ****************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Build WAL-G deps] *********************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Build and install WAL-G] **************************************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Generate conf file /var/lib/postgresql/.walg.json] ************************************************************************************************************************************************
changed: [10.116.0.3]

TASK [wal-g : Make sure that the cron package is installed] *****************************************************************************************************************************************************
ok: [10.116.0.3]

TASK [wal-g : Add WAL-G cron jobs] ******************************************************************************************************************************************************************************
changed: [10.116.0.3] => (item={'name': 'WAL-G: Create daily backup', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '00', 'hour': '3', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1"})
changed: [10.116.0.3] => (item={'name': 'WAL-G: Delete old backups', 'user': 'postgres', 'file': '/etc/cron.d/walg', 'minute': '30', 'hour': '6', 'day': '*', 'month': '*', 'weekday': '*', 'job': "[ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1"})
...

root@vitaliy-test2-pgcluster-pgnode01:~# su - postgres
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat .walg.json 
{
  "AWS_ACCESS_KEY_ID": "******",
  "AWS_SECRET_ACCESS_KEY": "*********",
  "AWS_ENDPOINT": "https://nyc3.digitaloceanspaces.com",
  "AWS_REGION": "nyc3",
  "AWS_S3_FORCE_PATH_STYLE": "True",
  "WALG_S3_PREFIX": "s3://backups-vitaliy-test2-pgcluster",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DELTA_MAX_STEPS": "6",
  "WALG_DOWNLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_CONCURRENCY": "1",
  "WALG_UPLOAD_DISK_CONCURRENCY": "1",
  "PGDATA": "/pgdata/16/vitaliy-test2-pgcluster",
  "PGHOST": "/var/run/postgresql",
  "PGPORT": "5432",
  "PGUSER": "postgres"
}
postgres@vitaliy-test2-pgcluster-pgnode01:~$ psql
psql (16.1 (Ubuntu 16.1-1.pgdg22.04+1))
Type "help" for help.

postgres=# show archive_command ;
  archive_command  
-------------------
 wal-g wal-push %p
(1 row)

postgres=# select pg_switch_wal();
 pg_switch_wal 
---------------
 0/1786D08
(1 row)

postgres=# select * from pg_stat_archiver;
 archived_count |    last_archived_wal     |      last_archived_time       | failed_count | last_failed_wal | last_failed_time |          stats_reset          
----------------+--------------------------+-------------------------------+--------------+-----------------+------------------+-------------------------------
              1 | 000000010000000000000001 | 2023-12-01 11:57:20.657624+00 |            0 |                 |                  | 2023-12-01 11:56:25.051441+00
(1 row)

postgres=# \q
postgres@vitaliy-test2-pgcluster-pgnode01:~$ cat /etc/cron.d/walg 
#Ansible: WAL-G: Create daily backup
00 3 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster > /var/log/postgresql/walg_backup.log 2>&1
#Ansible: WAL-G: Delete old backups
30 6 * * * postgres [ $(curl -s -o /dev/null -w '%{http_code}' http://10.116.0.3:8008) = '200' ] && wal-g delete retain FULL 4 --confirm > /var/log/postgresql/walg_delete.log 2>&1
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-push /pgdata/16/vitaliy-test2-pgcluster
INFO: 2023/12/01 11:57:39.466278 Couldn't find previous backup. Doing full backup.
INFO: 2023/12/01 11:57:39.477022 Calling pg_start_backup()
INFO: 2023/12/01 11:57:39.620068 Starting a new tar bundle
INFO: 2023/12/01 11:57:39.620108 Walking ...
INFO: 2023/12/01 11:57:39.620422 Starting part 1 ...
INFO: 2023/12/01 11:57:39.912292 Packing ...
INFO: 2023/12/01 11:57:39.913277 Finished writing part 1.
INFO: 2023/12/01 11:57:39.985622 Starting part 2 ...
INFO: 2023/12/01 11:57:39.985652 /global/pg_control
INFO: 2023/12/01 11:57:39.986493 Finished writing part 2.
INFO: 2023/12/01 11:57:39.986617 Calling pg_stop_backup()
INFO: 2023/12/01 11:57:40.030028 Starting part 3 ...
INFO: 2023/12/01 11:57:40.030118 backup_label
INFO: 2023/12/01 11:57:40.030134 tablespace_map
INFO: 2023/12/01 11:57:40.047375 Finished writing part 3.
INFO: 2023/12/01 11:57:40.125614 Wrote backup with name base_000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$ 
postgres@vitaliy-test2-pgcluster-pgnode01:~$ wal-g backup-list
name                          modified             wal_segment_backup_start
base_000000010000000000000003 2023-12-01T11:57:40Z 000000010000000000000003
postgres@vitaliy-test2-pgcluster-pgnode01:~$

passed

vitabaks commented 3 months ago

Test: Amazon Elastic Load Balancer (ELB)

Commit: 12b662206cbb7aa33e2aa64bfd9804673d4d8c4c aa0c104d4bc8ef8cb947ee4265e6979158bd4576

export AWS_ACCESS_KEY_ID=******
export AWS_SECRET_ACCESS_KEY=******

docker run --rm -it \
  --env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
  --env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
  vitabaks/postgresql_cluster:cloud \
    ansible-playbook deploy_pgcluster.yml --extra-vars \
      "ansible_user=ubuntu \
       cloud_provider='aws' \
       servers_count=3 \
       server_type=m5.xlarge \
       server_image=ami-0f1eef1771f0c6b68 \
       server_location=us-east-2 \
       volume_size=100 \
       ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\" \
       pgbouncer_install=true \
       postgresql_version=16 \
       patroni_cluster_name=vitabaks-pgcluster \
       cloud_load_balancer=true"

Result:

PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a unique temporary SSH key name] ***********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Generate a new temporary SSH key to access the server for deployment] *******************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Set variable: ssh_key_name and ssh_key_content] *****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'python3-pip' package is present on controlling host] ***********************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Ensure that 'boto3' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : AWS: Remove temporary SSH key 'ssh_key_tmp_rsqjych' from cloud (if any)] ****************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Add SSH key 'ssh_key_tmp_rsqjych' to cloud] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Gather information about default VPC] **********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Gather information about VPC subnet for default VPC] *******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: vpc_id] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : AWS: Create or modify Security Group] ***************************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : AWS: Create or modify EC2 instance] *****************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Wait for EC2 instance to be available via SSH] ******************************************************************************************************************************************
ok: [localhost] => (item=3.21.158.196)
ok: [localhost] => (item=3.144.85.6)
ok: [localhost] => (item=3.21.105.210)

TASK [cloud-resources : AWS: Create or modify Elastic Load Balancer (ELB)] **************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Show EC2 instance info] *****************************************************************************************************************************************************************
ok: [localhost] => (item=3.21.158.196) => {
    "msg": [
        "ID: i-00ad8e2bd22e27a9a",
        "Name: vitabaks-pgcluster-pgnode01",
        "Image: ami-0f1eef1771f0c6b68",
        "Type: m5.xlarge",
        "Volume Size: 100 GB",
        "Public IP: 3.21.158.196",
        "Private IP: 172.31.5.183"
    ]
}
ok: [localhost] => (item=3.144.85.6) => {
    "msg": [
        "ID: i-03c9c4021aa958e79",
        "Name: vitabaks-pgcluster-pgnode02",
        "Image: ami-0f1eef1771f0c6b68",
        "Type: m5.xlarge",
        "Volume Size: 100 GB",
        "Public IP: 3.144.85.6",
        "Private IP: 172.31.0.82"
    ]
}
ok: [localhost] => (item=3.21.105.210) => {
    "msg": [
        "ID: i-06e840e3b99754b83",
        "Name: vitabaks-pgcluster-pgnode03",
        "Image: ami-0f1eef1771f0c6b68",
        "Type: m5.xlarge",
        "Volume Size: 100 GB",
        "Public IP: 3.21.105.210",
        "Private IP: 172.31.7.30"
    ]
}

...

TASK [deploy-finish : Postgres list of users] *******************************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] ***************************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "dbs_result.stdout_lines": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] ********************************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitabaks-pgcluster (7328427680838806347) ---+----+-----------+",
        "| Member          | Host         | Role    | State     | TL | Lag in MB |",
        "+-----------------+--------------+---------+-----------+----+-----------+",
        "| ip-172-31-0-82  | 172.31.0.82  | Replica | streaming |  1 |         0 |",
        "| ip-172-31-5-183 | 172.31.5.183 | Leader  | running   |  1 |           |",
        "| ip-172-31-7-30  | 172.31.7.30  | Replica | streaming |  1 |         0 |",
        "+-----------------+--------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Postgres Cluster connection info] *********************************************************************************************************************************************************
ok: [172.31.5.183] => {
    "msg": [
        "+-------------------------------------------------------------------------+",
        "DNS (primary): internal-vitabaks-pgcluster-primary-1466392758.us-east-2.elb.amazonaws.com ",
        "DNS (replica): internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com ",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: ZBhh9eDRVUtj6f2tzqeIS8xKxfUvGHaK",
        "+-------------------------------------------------------------------------+"
    ]
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
172.31.0.82                : ok=112  changed=65   unreachable=0    failed=0    skipped=367  rescued=0    ignored=0   
172.31.5.183               : ok=130  changed=68   unreachable=0    failed=0    skipped=409  rescued=0    ignored=0   
172.31.7.30                : ok=112  changed=65   unreachable=0    failed=0    skipped=367  rescued=0    ignored=0   
localhost                  : ok=25   changed=6    unreachable=0    failed=0    skipped=157  rescued=0    ignored=0

Check primary:

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-primary-1466392758.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting     |  role   
------------------------+---------
 172.31.5.183,127.0.0.1 | primary
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-primary-1466392758.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting     |  role   
------------------------+---------
 172.31.5.183,127.0.0.1 | primary
(1 row)

Check replicas:

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.0.82,127.0.0.1 | replica
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.7.30,127.0.0.1 | replica
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.0.82,127.0.0.1 | replica
(1 row)

ubuntu@ip-172-31-5-183:~$ psql -h internal-vitabaks-pgcluster-replica-2105525505.us-east-2.elb.amazonaws.com -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
    current_setting    |  role   
-----------------------+---------
 172.31.7.30,127.0.0.1 | replica
(1 row)

passed

vitabaks commented 2 months ago

Test: DigitalOcean Load Balancer

commit: 55fd8e50f92092f6db52281052af087ffb0a630f 9f95b16d40ab93fb3ac7c7c1a086dc4135501896

export DO_API_TOKEN=*************

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider=digitalocean \
     server_count=3 \
     server_location=nyc1 \
     server_type=g-2vcpu-8gb \
     server_image=ubuntu-22-04-x64 \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\"
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitabaks-pgcluster \
     cloud_load_balancer=true"

Result:


PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] *******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'dopy' dependency is present on controlling host] **************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : DigitalOcean: Gather information about SSH keys] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Get fingerprint for all SSH keys] ****************************************************************************************************************************************
ok: [localhost] => (item=vitaliy)

TASK [cloud-resources : DigitalOcean: Gather information about VPC] ********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Extract ip_range from default VPC] *****************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create a tag 'vitabaks-pgcluster'] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify public firewall] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Postgres cluster firewall] ******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Droplet] ************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Set variable: droplet_result] **********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Block Storage] ******************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-storage)

TASK [cloud-resources : DigitalOcean: Attach Block Storage to Droplet] *****************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] *************************************************************************************************************************************************
ok: [localhost] => (item=67.205.177.108)
ok: [localhost] => (item=134.209.64.218)
ok: [localhost] => (item=157.230.92.31)

TASK [cloud-resources : Set variable: digital_ocean_load_balancer_port] ****************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: digital_ocean_load_balancer_target_port] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : DigitalOcean: Create or modify Load Balancer] ******************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : DigitalOcean: Gather information about load balancers] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Show Droplet info] *********************************************************************************************************************************************************************
ok: [localhost] => (item=67.205.177.108) => {
    "msg": [
        "ID: 403793886",
        "Name: vitabaks-pgcluster-pgnode01",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 67.205.177.108",
        "Private IP: 10.116.0.2"
    ]
}
ok: [localhost] => (item=134.209.64.218) => {
    "msg": [
        "ID: 403793973",
        "Name: vitabaks-pgcluster-pgnode02",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 134.209.64.218",
        "Private IP: 10.116.0.3"
    ]
}
ok: [localhost] => (item=157.230.92.31) => {
    "msg": [
        "ID: 403794046",
        "Name: vitabaks-pgcluster-pgnode03",
        "Image: Ubuntu 22.04 (LTS) x64",
        "Type: g-2vcpu-8gb",
        "Volume Size: 100 GB",
        "Public IP: 157.230.92.31",
        "Private IP: 10.116.0.4"
    ]
}

...

TASK [deploy-finish : Postgres list of users] ******************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] **************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "dbs_result.stdout_lines": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] *******************************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitabaks-pgcluster (7340630136958481870) -+-----------+----+-----------+",
        "| Member                      | Host       | Role    | State     | TL | Lag in MB |",
        "+-----------------------------+------------+---------+-----------+----+-----------+",
        "| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |",
        "| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |",
        "| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |",
        "+-----------------------------+------------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Postgres Cluster connection info] ********************************************************************************************************************************************************
ok: [10.116.0.2] => {
    "msg": [
        "+-------------------------------------------------------------------------+",
        "IP Address (primary): 143.244.212.53 ",
        "IP Address (replica): 167.172.12.113 ",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: 4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP",
        "+-------------------------------------------------------------------------+"
    ]
}

PLAY RECAP *****************************************************************************************************************************************************************************************************
10.116.0.2                 : ok=130  changed=65   unreachable=0    failed=0    skipped=414  rescued=0    ignored=0   
10.116.0.3                 : ok=111  changed=62   unreachable=0    failed=0    skipped=372  rescued=0    ignored=0   
10.116.0.4                 : ok=111  changed=62   unreachable=0    failed=0    skipped=372  rescued=0    ignored=0   
localhost                  : ok=25   changed=4    unreachable=0    failed=0    skipped=172  rescued=0    ignored=0

Check primary:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.2,127.0.0.1 | primary
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.2,127.0.0.1 | primary
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.2,127.0.0.1 | primary
(1 row)

Check replicas:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.4,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.3,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.4,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.3,127.0.0.1 | replica
(1 row)

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 167.172.12.113 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.4,127.0.0.1 | replica
(1 row)

Switchover:

root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+-----------+----+-----------+
| Member                      | Host       | Role    | State     | TL | Lag in MB |
+-----------------------------+------------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |
+-----------------------------+------------+---------+-----------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# patronictl switchover
Current cluster topology
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+-----------+----+-----------+
| Member                      | Host       | Role    | State     | TL | Lag in MB |
+-----------------------------+------------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | streaming |  1 |         0 |
+-----------------------------+------------+---------+-----------+----+-----------+
Primary [vitabaks-pgcluster-pgnode01]: 
Candidate ['vitabaks-pgcluster-pgnode02', 'vitabaks-pgcluster-pgnode03'] []: vitabaks-pgcluster-pgnode02
When should the switchover take place (e.g. 2024-02-28T13:42 )  [now]: 
Are you sure you want to switchover cluster vitabaks-pgcluster, demoting current leader vitabaks-pgcluster-pgnode01? [y/N]: y
2024-02-28 12:42:21.57382 Successfully switched over to "vitabaks-pgcluster-pgnode02"
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+---------+----+-----------+
| Member                      | Host       | Role    | State   | TL | Lag in MB |
+-----------------------------+------------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Replica | stopped |    |   unknown |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Leader  | running |  1 |           |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | running |  1 |         0 |
+-----------------------------+------------+---------+---------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# 
root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7340630136958481870) -+---------+----+-----------+
| Member                      | Host       | Role    | State   | TL | Lag in MB |
+-----------------------------+------------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.116.0.2 | Replica | running |  1 |         0 |
| vitabaks-pgcluster-pgnode02 | 10.116.0.3 | Leader  | running |  2 |           |
| vitabaks-pgcluster-pgnode03 | 10.116.0.4 | Replica | running |  1 |         0 |
+-----------------------------+------------+---------+---------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# 

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=4DJ3skC6R1yyE2ZEmYoqqG8YfeBlyUgP psql -h 143.244.212.53 -p 6432 -U postgres -c "select current_setting('listen_addresses'), case when pg_is_in_recovery() then 'replica' else 'primary' end as role"
   current_setting    |  role   
----------------------+---------
 10.116.0.3,127.0.0.1 | primary
(1 row)

passed

vitabaks commented 2 months ago

Test: Hetzner Load Balancer

Commit: 43022caeb072bb2389ea46f5af49c45404252aab 33c93a3562716ad6ae2b5e7c9c2b50d66fbe9b77

export HCLOUD_API_TOKEN=*********

ansible-playbook deploy_pgcluster.yml \
  --user=root --private-key=$HOME/.ssh/id_rsa --extra-vars \
    "cloud_provider=hetzner \
     server_count=3 \
     server_type=CCX13 \
     server_image=ubuntu-22.04 \
     server_location=ash \
     volume_size=100 \
     ssh_public_keys=\"$(cat $HOME/.ssh/id_rsa.pub)\"
     pgbouncer_install=true \
     postgresql_version=16 \
     patroni_cluster_name=vitabaks-pgcluster \
     cloud_load_balancer=true \
     database_public_access=false"

Result:


PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni")] ********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Include main variables] ***********************************************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Ensure that 'hcloud' dependency is present on controlling host] *************************************************************************************************************************
ok: [localhost -> 127.0.0.1]

TASK [cloud-resources : Hetzner Cloud: Gather information about SSH key 'vitaliy'] ******************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: ssh_key_names] ************************************************************************************************************************************************************
ok: [localhost] => (item=None)
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Gather information about network zones] **********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Extract network zone for server_location] ********************************************************************************************************************************
ok: [localhost] => (item=network_zone: us-east)

TASK [cloud-resources : Hetzner Cloud: Gather information about networks] ***************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a network 'postgres-cluster-network-us-east'] *********************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create a subnetwork in network 'postgres-cluster-network-us-east'] *******************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Set variable: server_network] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify public firewall] ****************************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify Postgres cluster firewall] ******************************************************************************************************************************
changed: [localhost]

TASK [cloud-resources : Hetzner Cloud: Create or modify server] *************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Add server to network 'postgres-cluster-network-us-east'] ****************************************************************************************************************
ok: [localhost] => (item=vitabaks-pgcluster-pgnode01)
ok: [localhost] => (item=vitabaks-pgcluster-pgnode02)
ok: [localhost] => (item=vitabaks-pgcluster-pgnode03)

TASK [cloud-resources : Hetzner Cloud: Create or modify volume] *************************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-pgnode01-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode02-storage)
changed: [localhost] => (item=vitabaks-pgcluster-pgnode03-storage)

TASK [cloud-resources : Wait for host to be available via SSH] **************************************************************************************************************************************************
ok: [localhost] => (item=5.161.59.224)
ok: [localhost] => (item=5.161.207.40)
ok: [localhost] => (item=5.161.181.46)

TASK [cloud-resources : Hetzner Cloud: Create or modify Load Balancer] ******************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Configure Load Balancer service] *****************************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Add Load Balancer to network 'postgres-cluster-network-us-east'] *********************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Disable public interface for Load Balancer] ******************************************************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Add servers to Load Balancer (use label_selector 'cluster=vitabaks-pgcluster')] ******************************************************************************************
changed: [localhost] => (item=vitabaks-pgcluster-primary)
changed: [localhost] => (item=vitabaks-pgcluster-replica)

TASK [cloud-resources : Hetzner Cloud: Gather information about Load Balancers] *********************************************************************************************************************************
ok: [localhost]

TASK [cloud-resources : Show Server info] ***********************************************************************************************************************************************************************
ok: [localhost] => (item=5.161.59.224) => {
    "msg": [
        "ID: 44673953",
        "Name: vitabaks-pgcluster-pgnode01",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.59.224",
        "Private IP: 10.0.1.1"
    ]
}
ok: [localhost] => (item=5.161.207.40) => {
    "msg": [
        "ID: 44673966",
        "Name: vitabaks-pgcluster-pgnode02",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.207.40",
        "Private IP: 10.0.1.2"
    ]
}
ok: [localhost] => (item=5.161.181.46) => {
    "msg": [
        "ID: 44674000",
        "Name: vitabaks-pgcluster-pgnode03",
        "Image: ubuntu-22.04",
        "Type: ccx13",
        "Volume Size: 100 GB",
        "Public IP: 5.161.181.46",
        "Private IP: 10.0.1.3"
    ]
}

...

TASK [deploy-finish : Postgres list of users] *******************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "users_result.stdout_lines": [
        "                              List of roles",
        " Role name  |                         Attributes                         ",
        "------------+------------------------------------------------------------",
        " pgbouncer  | ",
        " postgres   | Superuser, Create role, Create DB, Replication, Bypass RLS",
        " replicator | Replication"
    ]
}

TASK [deploy-finish : Postgres list of databases] ***************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "dbs_result.stdout_lines": [
        "                                                       List of databases",
        "   Name    |  Owner   | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   ",
        "-----------+----------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------",
        " postgres  | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | ",
        " template0 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        " template1 | postgres | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres          +",
        "           |          |          |                 |             |             |            |           | postgres=CTc/postgres",
        "(3 rows)"
    ]
}

TASK [deploy-finish : Postgres Cluster info] ********************************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "patronictl_result.stdout_lines": [
        "+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+",
        "| Member                      | Host     | Role    | State     | TL | Lag in MB |",
        "+-----------------------------+----------+---------+-----------+----+-----------+",
        "| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |",
        "| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |",
        "| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |",
        "+-----------------------------+----------+---------+-----------+----+-----------+"
    ]
}

TASK [deploy-finish : Postgres Cluster connection info] *********************************************************************************************************************************************************
ok: [10.0.1.1] => {
    "msg": [
        "+-------------------------------------------------------------------------+",
        "IP Address (primary): 10.0.1.4 ",
        "IP Address (replica): 10.0.1.5 ",
        "port: 6432 (pgbouncer)",
        "superuser: postgres",
        "password: kSVSQIWwUiZKY6N5iyJP9ikcaMHDhFXZ",
        "+-------------------------------------------------------------------------+"
    ]
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
10.0.1.1                   : ok=133  changed=69   unreachable=0    failed=0    skipped=408  rescued=0    ignored=0   
10.0.1.2                   : ok=113  changed=66   unreachable=0    failed=0    skipped=366  rescued=0    ignored=0   
10.0.1.3                   : ok=113  changed=66   unreachable=0    failed=0    skipped=366  rescued=0    ignored=0   
localhost                  : ok=29   changed=9    unreachable=0    failed=0    skipped=175  rescued=0    ignored=0

Check primary:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=kSVSQIWwUiZKY6N5iyJP9ikcaMHDhFXZ psql -h 10.0.1.4 -p 6432 -U postgres -d postgres -c "select pg_is_in_recovery()"
 pg_is_in_recovery 
-------------------
 f
(1 row)

Check replica:

root@vitabaks-pgcluster-pgnode01:~# PGPASSWORD=kSVSQIWwUiZKY6N5iyJP9ikcaMHDhFXZ psql -h 10.0.1.5 -p 6432 -U postgres -d postgres -c "select pg_is_in_recovery()"
 pg_is_in_recovery 
-------------------
 t
(1 row)

Switchover:

root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |
+-----------------------------+----------+---------+-----------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# patronictl switchover vitabaks-pgcluster
Current cluster topology
+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Leader  | running   |  1 |           |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Replica | streaming |  1 |         0 |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  1 |         0 |
+-----------------------------+----------+---------+-----------+----+-----------+
Primary [vitabaks-pgcluster-pgnode01]: 
Candidate ['vitabaks-pgcluster-pgnode02', 'vitabaks-pgcluster-pgnode03'] []: vitabaks-pgcluster-pgnode02    
When should the switchover take place (e.g. 2024-03-15T14:37 )  [now]: 
Are you sure you want to switchover cluster vitabaks-pgcluster, demoting current leader vitabaks-pgcluster-pgnode01? [y/N]: y
2024-03-15 13:37:33.96476 Successfully switched over to "vitabaks-pgcluster-pgnode02"
+ Cluster: vitabaks-pgcluster (7346582435914755030) ---------+----+-----------+
| Member                      | Host     | Role    | State   | TL | Lag in MB |
+-----------------------------+----------+---------+---------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Replica | stopped |    |   unknown |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Leader  | running |  1 |           |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | running |  1 |         0 |
+-----------------------------+----------+---------+---------+----+-----------+
root@vitabaks-pgcluster-pgnode01:~# 
root@vitabaks-pgcluster-pgnode01:~# patronictl list
+ Cluster: vitabaks-pgcluster (7346582435914755030) -----------+----+-----------+
| Member                      | Host     | Role    | State     | TL | Lag in MB |
+-----------------------------+----------+---------+-----------+----+-----------+
| vitabaks-pgcluster-pgnode01 | 10.0.1.1 | Replica | streaming |  2 |         0 |
| vitabaks-pgcluster-pgnode02 | 10.0.1.2 | Leader  | running   |  2 |           |
| vitabaks-pgcluster-pgnode03 | 10.0.1.3 | Replica | streaming |  2 |         0 |
+-----------------------------+----------+---------+-----------+----+-----------+

Check Replica load balancer

passed