search

vitabaks / postgresql_cluster

PostgreSQL High-Availability Cluster (based on "Patroni" and DCS "etcd" or "consul"). Automating with Ansible.
MIT License
1.29k stars 352 forks source link

hostvars is undefined (consul role) #437

Closed digitalcortex closed 9 months ago

digitalcortex commented 9 months ago

At the end of ansible-playbook deploy_pgcluster.yml command I receive an error for each node: "fatal: [XXX.XX.XXX.XXX]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'template'. Error was a <class 'ansible.errors.AnsibleFilterError'>, original message: Mandatory variable 'hostvars[' ']' not defined.. Mandatory variable 'hostvars[' ']' not defined."}".

Am I doing something wrong? I followed all instructions.

vitabaks commented 9 months ago

Please attach ansible log

digitalcortex commented 9 months ago 
PLAY [Deploy PostgreSQL HA Cluster (based on "Patroni" and "consul")] **********************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Include OS-specific variables] *******************************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [System information] ******************************************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "DO-Regular, count: 1, cores: 1",
        "Disk space total": "24.37 GB",
        "Kernel": "5.15.0-67-generic",
        "Memory": "0.95 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Droplet",
        "Virtualization type": "kvm"
    }
}
ok: [BBB.BB.BBB.BBB] => {
    "system_info": {
        "Architecture": "x86_64",
        "CPU model": "AMD EPYC 7713 64-Core Processor, count: 1, cores: 1",
        "Disk space total": "24.04 GB",
        "Kernel": "5.15.0-73-generic",
        "Memory": "0.95 GB",
        "OS": "Ubuntu 22.04",
        "Product name": "Compute Instance",
        "Virtualization type": "NA"
    }
}

TASK [pre-checks : Set max_connections from vars or use default] ***************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA] => (item={'option': 'max_connections', 'value': '500'})

TASK [pre-checks : PgBouncer | Calculate pool_size] ****************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'postgres', 'dbname': 'postgres', 'pool_parameters': ''})

TASK [pre-checks : PgBouncer | Calculate total pool_size] **********************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]

TASK [pre-checks : PgBouncer | Show total pool_size] ***************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA] => {
    "pgbouncer_total_pool_size": "20"
}

TASK [pre-checks : PostgreSQL | check that data directory "/var/lib/postgresql/15/main" is not initialized] ********************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Update apt cache] ********************************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Make sure the gnupg and apt-transport-https packages are present] ********************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Make sure that the iproute is installed] *********************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

PLAY [Configure etcd Cluster and System Settings] ******************************************************************************************************************************************************************
[WARNING]: flush_handlers task does not support when conditional

PLAY [consul.yml | Consul Playbook] ********************************************************************************************************************************************************************************

TASK [Check if the consul role requirements (ansible.utils) are installed] *****************************************************************************************************************************************
ok: [localhost]

PLAY [consul.yml | Configure Consul instances] *********************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Include OS-specific variables] *******************************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [Update apt cache] ********************************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Make sure the gnupg and apt-transport-https packages are present] ********************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Make sure the python3-pip package are present] ***************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [Make sure the unzip package are present on the control host] *************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [hostname : Change hostname] **********************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [hostname : Change hostname in /etc/hosts] ********************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB] => (item=None)
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA] => (item=None)
ok: [AAA.AAA.AAA.AAA]

TASK [resolv_conf : Make sure /etc/resolv.conf exists] *************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [resolv_conf : Add DNS server(s) into /etc/resolv.conf] *******************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB] => (item=127.0.0.1)
ok: [AAA.AAA.AAA.AAA] => (item=127.0.0.1)

TASK [sysctl : Build a sysctl_conf dynamic variable] ***************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA] => (item=balancers)
ok: [AAA.AAA.AAA.AAA] => (item=consul_instances)
ok: [BBB.BB.BBB.BBB] => (item=balancers)
ok: [AAA.AAA.AAA.AAA] => (item=etcd_cluster)
ok: [BBB.BB.BBB.BBB] => (item=consul_instances)
ok: [BBB.BB.BBB.BBB] => (item=etcd_cluster)
ok: [AAA.AAA.AAA.AAA] => (item=master)
ok: [AAA.AAA.AAA.AAA] => (item=postgres_cluster)
ok: [BBB.BB.BBB.BBB] => (item=postgres_cluster)
ok: [BBB.BB.BBB.BBB] => (item=replica)

TASK [sysctl : Setting kernel parameters] **************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'net.ipv4.ip_nonlocal_bind', 'value': '1'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'net.ipv4.ip_local_port_range', 'value': '10000 65535'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'net.core.netdev_max_backlog', 'value': '10000'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'net.ipv4.tcp_max_syn_backlog', 'value': '8192'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'net.core.somaxconn', 'value': '65535'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'net.ipv4.tcp_tw_reuse', 'value': '1'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [AAA.AAA.AAA.AAA] => (item={'name': 'vm.overcommit_memory', 'value': '2'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'vm.swappiness', 'value': '1'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'vm.swappiness', 'value': '1'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'vm.min_free_kbytes', 'value': '102400'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'vm.dirty_expire_centisecs', 'value': '1000'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'vm.dirty_background_bytes', 'value': '67108864'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'vm.dirty_bytes', 'value': '536870912'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'vm.zone_reclaim_mode', 'value': '0'})
ok: [BBB.BB.BBB.BBB] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'kernel.numa_balancing', 'value': '0'})
changed: [AAA.AAA.AAA.AAA] => (item={'name': 'kernel.sched_autogroup_enabled', 'value': '0'})

TASK [consul : Make sure the python3-pip package are present on controlling host] **********************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Install netaddr dependency on controlling host (with --user)] ***************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Install netaddr dependency on controlling host (virtualenv)] ****************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Define supported *nix distributions] ****************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [consul : Check if unzip is installed on control host] ********************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Include OS-specific variables] **********************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA] => (item=/root/postgresql_cluster/roles/consul/vars/Debian.yml)
ok: [BBB.BB.BBB.BBB] => (item=/root/postgresql_cluster/roles/consul/vars/Debian.yml)

TASK [consul : Include NIX tasks] **********************************************************************************************************************************************************************************
included: /root/postgresql_cluster/roles/consul/tasks/nix.yml for AAA.AAA.AAA.AAA, BBB.BB.BBB.BBB

TASK [consul : Expose advertise_address(_wan) datacenter and node_role as facts] ***********************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [consul : Read bootstrapped state] ****************************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [consul : Add Consul group] ***********************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
changed: [AAA.AAA.AAA.AAA]

TASK [consul : Add Consul user] ************************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
changed: [AAA.AAA.AAA.AAA]

TASK [consul : Configuration and data directories] *****************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB] => (item=/etc/consul)
changed: [AAA.AAA.AAA.AAA] => (item=/etc/consul)
ok: [BBB.BB.BBB.BBB] => (item=/etc/consul/conf.d)
changed: [AAA.AAA.AAA.AAA] => (item=/etc/consul/conf.d)
ok: [BBB.BB.BBB.BBB] => (item=/var/lib/consul)
changed: [AAA.AAA.AAA.AAA] => (item=/var/lib/consul)

TASK [consul : Run directory] **************************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
changed: [AAA.AAA.AAA.AAA]

TASK [consul : Verify binary path] *********************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]
ok: [AAA.AAA.AAA.AAA]

TASK [consul : Check for existing Consul binary] *******************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [consul : Get current Consul version] *************************************************************************************************************************************************************************
ok: [BBB.BB.BBB.BBB]

TASK [consul : Calculate whether to install consul binary] *********************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [consul : Install OS packages and consul - locally] ***********************************************************************************************************************************************************
included: /root/postgresql_cluster/roles/consul/tasks/install.yml for AAA.AAA.AAA.AAA

TASK [consul : Install OS packages] ********************************************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA]

TASK [consul : Read package checksum file] *************************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Read package checksum] ******************************************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Check Consul package file] **************************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Create Temporary Directory for Extraction] **********************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Unarchive Consul package] ***************************************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Install Consul] *************************************************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA]

TASK [consul : Cleanup] ********************************************************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA -> 127.0.0.1]

TASK [consul : Generate gossip encryption key] *********************************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA]

TASK [consul : Write key locally to share with other nodes] ********************************************************************************************************************************************************
changed: [AAA.AAA.AAA.AAA -> localhost]

TASK [consul : Read gossip encryption key for servers that require it] *********************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA]
ok: [BBB.BB.BBB.BBB]

TASK [consul : Delete gossip encryption key file] ******************************************************************************************************************************************************************
ok: [AAA.AAA.AAA.AAA -> localhost]

TASK [consul : Create configuration] *******************************************************************************************************************************************************************************
fatal: [AAA.AAA.AAA.AAA]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'template'. Error was a <class 'ansible.errors.AnsibleFilterError'>, original message: Mandatory variable 'hostvars[' ']'  not defined.. Mandatory variable 'hostvars[' ']'  not defined."}
fatal: [BBB.BB.BBB.BBB]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'template'. Error was a <class 'ansible.errors.AnsibleFilterError'>, original message: Mandatory variable 'hostvars[' ']'  not defined.. Mandatory variable 'hostvars[' ']'  not defined."}

NO MORE HOSTS LEFT *************************************************************************************************************************************************************************************************

PLAY RECAP *********************************************************************************************************************************************************************************************************
AAA.AAA.AAA.AAA            : ok=52   changed=13   unreachable=0    failed=1    skipped=124  rescued=0    ignored=0   
BBB.BB.BBB.BBB             : ok=32   changed=0    unreachable=0    failed=1    skipped=116  rescued=0    ignored=0   
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
vitabaks commented 9 months ago

@digitalcortex it is not yet clear why hostvars[' '] is empty can you attach an archive of your playbook so that I can analyze all the changes that you have made?

naquad commented 9 months ago

Same issue here at the same step. Tried to debug but no luck. This is my setup + Vagrant environment. Git directory included for the change tracking (very few).

vagrant up also regenerates the inventory.

digitalcortex commented 9 months ago

@digitalcortex it is not yet clear why hostvars[' '] is empty can you attach an archive of your playbook so that I can analyze all the changes that you have made?

I only changed the IP addresses in the inventory file, nothing else

vitabaks commented 9 months ago

try installing the netaddr package on the machine you are running the playbook from

pip3 install netaddr
naquad commented 9 months ago

It is installed.

vitabaks commented 9 months ago

let me know if you can figure it out before me. Just in case, here is a link to the consul role project, maybe there is a solution there

https://github.com/ansible-community/ansible-consul

vitabaks commented 9 months ago

Fix here https://github.com/vitabaks/postgresql_cluster/pull/440