search

robertdebock / ansible-role-vault

Install Hashicorp Vault, either a package or a binary.
https://robertdebock.nl/
Apache License 2.0
33 stars 13 forks source link

Unable to unseal followers, 2 of 3 passes are successful, 1 is not successfull. #8

Closed nartykalyLV closed 2 years ago

nartykalyLV commented 2 years ago

Describe the bug

I'm trying to write IaC using terraform, ansible to setup vault cluster. I found that playbook errors, playbook run 2 of 3 passes are successful, 1 is not successful.

make provision
. config.sh
ssh-add $PVT_KEY
Identity added: /home/lindenvalley/.ssh/id_ed25519 (lindenvalley@worker)
ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_ROLES_PATH=~/git/ansible/roles ansible-playbook -i inventory playbooks/vault.yaml

PLAY [Provision server] ********************************************************************************************************************

TASK [Wait before cloudinit is finished] ***************************************************************************************************
ok: [vault-mhsmxqiqxmfs]
ok: [vault-oiaqvsajgslo]
ok: [vault-gbkylfpknrvp]

PLAY [prepare] *****************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************
ok: [vault-mhsmxqiqxmfs]
ok: [vault-oiaqvsajgslo]
ok: [vault-gbkylfpknrvp]

TASK [robertdebock.core_dependencies : install packages] ***********************************************************************************
changed: [vault-gbkylfpknrvp]
changed: [vault-oiaqvsajgslo]
changed: [vault-mhsmxqiqxmfs]

TASK [robertdebock.core_dependencies : try to install pip packages] ************************************************************************
changed: [vault-gbkylfpknrvp]
changed: [vault-mhsmxqiqxmfs]
changed: [vault-oiaqvsajgslo]

TASK [robertdebock.core_dependencies : flush handlers] *************************************************************************************

RUNNING HANDLER [robertdebock.core_dependencies : gather facts] ****************************************************************************
ok: [vault-gbkylfpknrvp]
ok: [vault-mhsmxqiqxmfs]
ok: [vault-oiaqvsajgslo]

TASK [robertdebock.hashicorp : test if hashicorp_installation_method is set correctly] *****************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.hashicorp : test if hashicorp_products is set correctly] ****************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.hashicorp : test if item in hashicorp_products is set correctly (package)] **********************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.hashicorp : test if item in hashicorp_products is set correctly (manual)] ***********************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.hashicorp : install repository for RedHat] ******************************************************************************
skipping: [vault-gbkylfpknrvp]
skipping: [vault-mhsmxqiqxmfs]
skipping: [vault-oiaqvsajgslo]

TASK [robertdebock.hashicorp : install apt key for Debian] *********************************************************************************
changed: [vault-gbkylfpknrvp]
changed: [vault-mhsmxqiqxmfs]
changed: [vault-oiaqvsajgslo]

TASK [robertdebock.hashicorp : install repository for Debian] ******************************************************************************
changed: [vault-oiaqvsajgslo]
changed: [vault-gbkylfpknrvp]
changed: [vault-mhsmxqiqxmfs]

TASK [robertdebock.hashicorp : install hashicorp product using package] ********************************************************************
skipping: [vault-gbkylfpknrvp]
skipping: [vault-mhsmxqiqxmfs]
skipping: [vault-oiaqvsajgslo]

TASK [robertdebock.hashicorp : install hashicorp product manually] *************************************************************************
skipping: [vault-gbkylfpknrvp]
skipping: [vault-mhsmxqiqxmfs]
skipping: [vault-oiaqvsajgslo]

TASK [Update /etc/hosts] *******************************************************************************************************************
changed: [vault-gbkylfpknrvp]
changed: [vault-oiaqvsajgslo]
changed: [vault-mhsmxqiqxmfs]

PLAY [Assemble Vault cluster] **************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************
ok: [vault-oiaqvsajgslo]
ok: [vault-gbkylfpknrvp]
ok: [vault-mhsmxqiqxmfs]

TASK [robertdebock.vault : test if vault_owner is set correctly] ***************************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_group is set correctly] ***************************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_disable_clustering is set correctly] **************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_cluster_addr is set correctly] ********************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_api_addr is set correctly] ************************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_plugin_directory is set correctly] ****************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_storages is set correctly] ************************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if item in vault_storages is set correctly] ****************************************************************
ok: [vault-gbkylfpknrvp -> localhost] => (item=raft)

TASK [robertdebock.vault : test if vault_listeners is set correctly] ***********************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if item in vault_listeners is set correctly] ***************************************************************
ok: [vault-gbkylfpknrvp -> localhost] => (item=tcp)
ok: [vault-gbkylfpknrvp -> localhost] => (item=tcp)

TASK [robertdebock.vault : test if item in vault_listeners is set correctly when tls_disable is yes] ***************************************
skipping: [vault-gbkylfpknrvp] => (item=tcp) 
skipping: [vault-gbkylfpknrvp] => (item=tcp) 

TASK [robertdebock.vault : test if vault_ui is set correctly] ******************************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_key_shares is set correctly] **********************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_key_threshold is set correctly] *******************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_key_shares and vault_key_threshold are set correctly] *********************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_show_unseal_information is set correctly] *********************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_unseal_keys is set correctly] *********************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_disable_mlock is set correctly] *******************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_make_backup is set correctly] *********************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_backup_path is set correctly] *********************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_namespace is set correctly] ***********************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_kv_secrets is set correctly] **********************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if item vault_kv_secrets is set correctly] *****************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if item.cas vault_kv_secrets is set correctly] *************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_kv_max_versions is set correctly] *****************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_kv_cas_required is set correctly] *****************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_kv_delete_version_after is set correctly] *********************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_license is set correctly] *************************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_log_level is set correctly] ***********************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_max_lease_ttl is set correctly] *******************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_default_lease_ttl is set correctly] ***************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : test if vault_transit is set correctly] *************************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_disable_cache is set correctly] *******************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_disable_clustering is "true" for raft] ************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : test if vault_store_root_token is set correctly] ****************************************************************
ok: [vault-gbkylfpknrvp -> localhost]

TASK [robertdebock.vault : run shared tasks] ***********************************************************************************************
included: /home/lindenvalley/git/ansible/roles/robertdebock.vault/tasks/shared.yml for vault-gbkylfpknrvp, vault-mhsmxqiqxmfs, vault-oiaqvsajgslo

TASK [robertdebock.vault : install vault] **************************************************************************************************
changed: [vault-oiaqvsajgslo]
changed: [vault-mhsmxqiqxmfs]
changed: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : create vault storage path] **************************************************************************************
changed: [vault-gbkylfpknrvp] => (item=/vault/data)
changed: [vault-oiaqvsajgslo] => (item=/vault/data)
changed: [vault-mhsmxqiqxmfs] => (item=/vault/data)

TASK [robertdebock.vault : make plugin directory] ******************************************************************************************
changed: [vault-gbkylfpknrvp]
changed: [vault-oiaqvsajgslo]
changed: [vault-mhsmxqiqxmfs]

TASK [robertdebock.vault : select leader if vault_leader is unset] *************************************************************************
skipping: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : create /etc/vault.d] ********************************************************************************************
ok: [vault-mhsmxqiqxmfs]
ok: [vault-gbkylfpknrvp]
ok: [vault-oiaqvsajgslo]

TASK [robertdebock.vault : place /etc/vaultd.d/config.hcl] *********************************************************************************
changed: [vault-oiaqvsajgslo]
changed: [vault-mhsmxqiqxmfs]
changed: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : place vault license] ********************************************************************************************
skipping: [vault-gbkylfpknrvp]
skipping: [vault-mhsmxqiqxmfs]
skipping: [vault-oiaqvsajgslo]

TASK [robertdebock.vault : start vault] ****************************************************************************************************
changed: [vault-gbkylfpknrvp]
changed: [vault-mhsmxqiqxmfs]
changed: [vault-oiaqvsajgslo]

TASK [robertdebock.vault : check status of vault] ******************************************************************************************
ok: [vault-mhsmxqiqxmfs]
ok: [vault-oiaqvsajgslo]
ok: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : save vault_status] **********************************************************************************************
ok: [vault-gbkylfpknrvp]
ok: [vault-mhsmxqiqxmfs]
ok: [vault-oiaqvsajgslo]

TASK [robertdebock.vault : safe VAULT_ADDR /etc/environment] *******************************************************************************
changed: [vault-mhsmxqiqxmfs]
changed: [vault-oiaqvsajgslo]
changed: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : run leader tasks] ***********************************************************************************************
included: /home/lindenvalley/git/ansible/roles/robertdebock.vault/tasks/leader.yml for vault-gbkylfpknrvp, vault-mhsmxqiqxmfs, vault-oiaqvsajgslo

TASK [robertdebock.vault : initialize vault on leader] *************************************************************************************
changed: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : save vault_init_output for leader] ******************************************************************************
ok: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : show unseal keys for leader] ************************************************************************************
ok: [vault-gbkylfpknrvp] => {
    "msg": [
        "ZVOlGJXBPI0zN3845ncd/hU05fevreXtp4BDQmACiZZx",
        "FDYxqeRl6xamHS7hSQglvQ8sngwm06n60s0n2a4w9XRn",
        "zHVIUF6OdM4O8TtPXndMWz6RGAIp9oyxyW7QIpaCg3MR",
        "RLH+tIXXJsE9PETNsvJXv/nlZ9iLtYKNnBRLkhYkToIy",
        "lylefZkM5HqqVXohEkju3fTdIsAiojQ6EQGo5KSMUNmx"
    ]
}

TASK [robertdebock.vault : show root token for leader] *************************************************************************************
ok: [vault-gbkylfpknrvp] => {
    "msg": "s.gEjGr9UgabjHzK7JcvRbebzx"
}

TASK [robertdebock.vault : save login token to /root/.vault-token] *************************************************************************
changed: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : unseal vault leader] ********************************************************************************************
changed: [vault-gbkylfpknrvp] => (item=None)
changed: [vault-gbkylfpknrvp] => (item=None)
changed: [vault-gbkylfpknrvp] => (item=None)
changed: [vault-gbkylfpknrvp] => (item=None)
changed: [vault-gbkylfpknrvp] => (item=None)
changed: [vault-gbkylfpknrvp]

TASK [robertdebock.vault : make a snapshot] ************************************************************************************************
skipping: [vault-gbkylfpknrvp]
skipping: [vault-mhsmxqiqxmfs]
skipping: [vault-oiaqvsajgslo]

TASK [robertdebock.vault : run follower tasks] *********************************************************************************************
included: /home/lindenvalley/git/ansible/roles/robertdebock.vault/tasks/follower.yml for vault-gbkylfpknrvp, vault-mhsmxqiqxmfs, vault-oiaqvsajgslo

TASK [robertdebock.vault : join follower to leader] ****************************************************************************************
ok: [vault-gbkylfpknrvp]
FAILED - RETRYING: join follower to leader (5 retries left).
FAILED - RETRYING: join follower to leader (5 retries left).
FAILED - RETRYING: join follower to leader (4 retries left).
FAILED - RETRYING: join follower to leader (4 retries left).
ok: [vault-mhsmxqiqxmfs]
ok: [vault-oiaqvsajgslo]

TASK [robertdebock.vault : unseal vault follower] ******************************************************************************************
ok: [vault-gbkylfpknrvp] => (item=None)
ok: [vault-mhsmxqiqxmfs] => (item=None)
ok: [vault-oiaqvsajgslo] => (item=None)
ok: [vault-gbkylfpknrvp] => (item=None)
ok: [vault-mhsmxqiqxmfs] => (item=None)
ok: [vault-oiaqvsajgslo] => (item=None)
ok: [vault-gbkylfpknrvp] => (item=None)
failed: [vault-oiaqvsajgslo] (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
failed: [vault-mhsmxqiqxmfs] (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
ok: [vault-gbkylfpknrvp] => (item=None)
ok: [vault-oiaqvsajgslo] => (item=None)
ok: [vault-mhsmxqiqxmfs] => (item=None)
ok: [vault-gbkylfpknrvp] => (item=None)
ok: [vault-gbkylfpknrvp]
ok: [vault-oiaqvsajgslo] => (item=None)
fatal: [vault-oiaqvsajgslo]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
[vault-oiaqvsajgslo] TASK: robertdebock.vault : unseal vault follower (debug)> ok: [vault-mhsmxqiqxmfs] => (item=None)
p
***SyntaxError:SyntaxError('unexpected EOF while parsing', ('<string>', 0, 0, ''))
[vault-oiaqvsajgslo] TASK: robertdebock.vault : unseal vault follower (debug)> h

Documented commands (type help <topic>):
========================================
EOF  c  continue  h  help  p  pprint  q  quit  r  redo  u  update_task

[vault-oiaqvsajgslo] TASK: robertdebock.vault : unseal vault follower (debug)> c
fatal: [vault-mhsmxqiqxmfs]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
[vault-mhsmxqiqxmfs] TASK: robertdebock.vault : unseal vault follower (debug)> p
***SyntaxError:SyntaxError('unexpected EOF while parsing', ('<string>', 0, 0, ''))
[vault-mhsmxqiqxmfs] TASK: robertdebock.vault : unseal vault follower (debug)> c

NO MORE HOSTS LEFT *************************************************************************************************************************

PLAY RECAP *********************************************************************************************************************************
vault-gbkylfpknrvp         : ok=54   changed=14   unreachable=0    failed=0    skipped=20   rescued=0    ignored=0   
vault-mhsmxqiqxmfs         : ok=22   changed=11   unreachable=0    failed=1    skipped=5    rescued=0    ignored=0   
vault-oiaqvsajgslo         : ok=22   changed=11   unreachable=0    failed=1    skipped=5    rescued=0    ignored=0   

make: *** [Makefile:24: provision] Error 2

Playbook

Please paste the playbook you are using. (Consider requirements.yml and optionally the command you've invoked.)

---
# File: site.yml - Example Consul site playbook
- name: Provision server
  hosts: all
  remote_user: nartykaly
  gather_facts: False
  no_log: false
  tasks:
    - name: Wait before cloudinit is finished
      wait_for:
        path: /var/lib/cloud/instance/boot-finished

- name: prepare
  hosts: vault_instances
  become: true
  become_method: sudo
  roles:
    - role: robertdebock.core_dependencies
    - role: robertdebock.hashicorp
  tasks:
    - name: Update /etc/hosts
      blockinfile:
        path: /etc/hosts
        block: |
          {% for host in groups['all'] %}
          {{ hostvars[host].ansible_host }} {{ host }}
          {% endfor %}

- name: Assemble Vault cluster
  hosts: vault_instances
  any_errors_fatal: true
  become: true
  become_user: root
  gather_facts: true
  debugger: on_failed
  roles:
    - robertdebock.vault
  vars:
    vault_show_unseal_information: yes
    vault_store_root_token: yes
    vault_make_backup: no
    vault_leader: "{{ hostvars[groups['vault_instances'][0]].ansible_host }}"
    vault_listeners:
      - name: tcp
        address: "127.0.0.1:8200"
        cluster_address: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8201"
        tls_disable: "true"

      - name: tcp
        address: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8200"
        cluster_address: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8201"
        tls_disable: "true"
    vault_log_level: "debug"
    vault_disable_clustering: "false"
    vault_api_addr: "http://{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8200"

Output

Show at least the error, possible related output, maybe just all the output.

Environment

nartykalyLV commented 2 years ago

Changed playbook to

---
# File: site.yml - Example Consul site playbook
- name: Provision server
  hosts: all
  remote_user: nartykaly
  gather_facts: False
  no_log: false
  tasks:
    - name: Wait before cloudinit is finished
      wait_for:
        path: /var/lib/cloud/instance/boot-finished

- name: prepare
  hosts: vault_instances
  become: true
  become_method: sudo
  roles:
    - role: robertdebock.core_dependencies
    - role: robertdebock.hashicorp
  tasks:
    - name: Update /etc/hosts
      blockinfile:
        path: /etc/hosts
        block: |
          {% for host in groups['all'] %}
          {{ hostvars[host].ansible_host }} {{ host }}
          {% endfor %}

- name: Assemble Vault cluster
  hosts: vault_instances
  any_errors_fatal: true
  become: true
  become_user: root
  gather_facts: true
  debugger: on_failed
  roles:
    - robertdebock.vault
  vars:
    vault_show_unseal_information: yes
    vault_store_root_token: yes
    vault_make_backup: no
    vault_leader: "{{ hostvars[groups['vault_instances'][0]].ansible_host }}"
    vault_listeners:
      - name: tcp
        address: "127.0.0.1:8200"
        cluster_address: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8201"
        tls_disable: "true"

      - name: tcp
        address: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8200"
        cluster_address: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8201"
        tls_disable: "true"
    vault_log_level: "debug"
    vault_disable_clustering: "false"
    vault_api_addr: "http://{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8200"
    vault_cluster_addr: "http://{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:8201"

and it is works now.